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FOREWORD ■ 

The John E. Fogai ty ' International Center for Advanced Study in 
th^ Health SdenceS is a federaHy established focal point for sharing 
America's: health research with more than 48 other nations of the 
world. The Center 'also functions as a -forum for drawim^foge'ther 
tf)<;^j|kp^s arid resources of other countries^and en^agihg tKem in 
way^at bear on the mahy anc^ varied health problems of th,e' United 
Swtes. Since 1968 wh?n it was designated a memorial to Congressman 
John E_. Fogarty of Rhode Island, th^ Center hVs pursuea- Mr. 
pQgarty's' vision of international scientific cooperation in the interests 
of mankind/ / 

The Fogarty Center s programs, developed by medical. Jiealth,:jind 
administrative professionals-, promote advanced study in biomedical 
and reflated scieoccs with »view toward improving health and physical 
wcltrbcing through practical means. Under the aegis of the Confer- 
ciiji ynd Seminar Pro^iiam of the Center, scientists are bro^ght 
f^Sptie.r'- fo,> study -aiid discussion of topics such as liver disease 
diabt'tes iiiellitus, and chronic childho.od illne.ss. The (;eographic 
• Health Studies-Program sponsors analyses of the health care syst<;m 
and medical research of other countries to determine how their 
. expeMeiices might serve as models foi' health cai>e in the United 
.: Ijrvj^^atfjs. \ft)i eover, scientists who have been singled out for their 
- -^heiYal c;oiiceriis by their own countries can aj^l^ for fellowship 
, awbrd> through the Schtilars and Fellowships Program. I he.se awards 
V.^__piij^ble th«^i to work clo.sely with U.S. .scientists at NIH or at academic 
in.stiiutions throughout the country. In like manner,' American 
researchers are sponsored for .study abroad. 

During the years of -its formation and growth, thfe Fogaity Inter- 
national Center has been dedicated to the use of sciencejbr peaceftil 



purpo.ses. for the^good o_|; all .human beings. To. this end, the pres^ 
monograph has been prepared, in cooperation with the American 
C()llege of Preventive Medicim^ th'e Pan .American Community 
Health As.socialion; the Bureau of Community Health Services, 
Health Services Admiiii.sti ation; the Office of Child Development 
and the Offic? of Human Development, DHEW; and the Natio/al 
Institute of Child Health and Human Development. , ■ I 

Thi.s monograph derives from a conference sponifeied by the 
Fogaity Center which brought together 65 experts in evaluatioiiT^ 
organizers of healthcare systems, difect^'^^J child health programs 



. N- ■ ■ 

university-aff iliated ediicatprs. social ^ciefttists. economists, policymak- 
ers, and consumer .advocates frdm the United States.-and Latin 
America. t)f primary concern was the need for a current evaluation 
of jif^tern^il and child health services delivery systems and t,he 
implications of this eviiluation for heal tfi professioniil education in 
the Americas. It is hoped that the discussions in this yolume wHI form 
a compreheni^ive report about a variety o£ issues relat^ to the health 
of the world's children. 



Mu.o D. LEAvrtT, Jr., m,d. . 
Director 

Fogarty International Center 
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WELCOMING REMARKS 

Milo Drleavitt Jr. 

In September 1975, an agreement was reached among officers of 
the Fogarty International Center, the American College of Preventive 
Medicine, and the Pan American Community Health Association to 
hold a workshop on child health care in the Americas that would 
focus primarily on methodologies for evaluating health services to 
children. The focus and target population were chosen because of 
the. growing interest and activities in different countries in the 
Americas in evaluation of these services. This has provided an 
excellent opportunity for an international exchange of information; 

Dr. SamuePJ. "Bosch, of the American College of Preventive 
Medicine, and Associate.jProfessor of Community Medicine, Mount 
Sinai School of Medicine, New York, and Dr. ^Antonio Ordoriez-Plaja, 
representing the , Pan American Community Health Asspciation, and 
Director of the Colombian National Institute of Family Welfare, were 
given the responsibility for planning and chairing the workshop. Dr. 
Jaime Arias, of the Colombian Department of Public Health,, has 
undertaken the important tas^k^ coordinating the reporting on 
evaluation projects ih different coi^tries as well as the d^elopment 
of that information for presentation and analysis at this workshop. 

Several agencies within the Department of Health, Education, and 
Welfare have program interest in this n^atter and are cosponsors of' 
this workshop. In addition, a number of ^heir key scientific staff are 
participants. The Rogarty Center staff has Worked enthusiastically 
with the planning committee to put together what we hope will be a 
successful, informative conference. 
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WELCOMING REMARKS ■ 

. Samuel J. Bosch 

I would like to welcome you to this meeting in the namt of the 
American College of Preventive Medicine, The College, though 
basically a U.S. organization, has always recognized the importance of 
having close ties with profe.ssionals dedicated to preventive medicine 
in other>parts of the vCorld. Indicative of^ties, interest, and feelings of 
collegiality is, the fact tha't the College recently established a new 
category of internationi^ membership for physiciarYs in the specialty 
of preventive medicine. In sponsoring meetings suci^^as this, we are 
. looking for new and better ways to develop knowledge about issues 

that are of common interest around the world. ^ 

This particular meeting has been planned and organized by 
representatives of the sponso^ii^g institutions. The planning has 
defined the following goals for the meeting: 

— To establish channels of communication between health care 
evaluatoi:s from different disciplines and .from different countries 
in the Americas. 

\^ — To promote an exchange of information and experiences in 
evaluation techniques: comparing approaches, methods, needed 
resourcesTdifficulties, achievements, and failures. 

Evaluation of health care was selected as a topic because it was 
recognized that this problem is com#non to* many countries, and gne 
about which we still have very limited knowledge. The planning also 
stressed the fact that, language and cultural barriers can be minimized 
through this topic. The team cjiose child care as the main focus 
because solving child health problems is a recognized priority in most 
ot-fhe .American countries. 

We hope that this conference will provide each of us an opportu- 
nity for analysis of mir own experience in evaluatioi> and for thought 
about its amplications for each of our countries. ; 
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CHILD HEALTH IN THE AMERICAS: A HISTORICAL 
AND GLOBAL PERSPECTIVE 

Introduction * ^^^^ ' ^ . ' 

Child health in the Americas is a reflection of the quality of health 
services provided. The he^th of children depends, of courseT^on 
other things as well. The socioeconomic status of their families is 
surely one of the most important factors. Many, thoughtful observers 
believe, p fact, that the health stalMs of children— especially in the 
first few years of life — one of the best indicators of a country's 
social and economic development. Those of us Who are concerned, 
professionally or* otherwise, wjth the welfare of children in the 
Americas know that the available data indicate extreme variation in 
the status of children in different countries of our hemisphere. We 
are fortunate in that this information/ uneven as it may be in some 
respects, is far better in this hemisphere than in most other regions 
of the world. 

The Pan American Health Organization (PAHO) deserves credit 
for much of this information; among the most revealing of the 
PAHO studies is Patterns of Mortality in Childhood (Puffer and Serrano 
1971). Mortality, of course, is a negative indicator of ^health, but it is 
the best we have and it is exti;emely useful. By examining the trends 
in mortality patterns over the last few years we can at least estimate 
the progress that has been made in the status of child health, in the 
Americas. By studying the available data concerning certain other 
specific, major problems that afflictxthe children in this hemisphere, 
we can get some idea of where we should concentrate our efforts to 
improve child health in this region. Finally, by placing this study in 
both a historical and a global perspective, we can derive some 
estimates of what is likely to happen if present trends continue. 

Trends in Childhood Mortality in the Western Hemisphere 

Two simple but revealing indicators of the status of child health 
are the proportion of all deaths that occur in children under age 5, 
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Figure \, Percentage of deaths under age 5. Western Hemisphere, 1972 
V [Redrawn from PAHO 1974] 



and infant mortality rates. Figure 1 shows the cl^ahges in the 
proportion of deaths occurring in children under ages 1 and 5 in 
Northern America, Middle America, and South America since 1964. 
We know from ^Northern American data» as well as from wmilar , 
Europcap data, that when children are well nourished and adequately 
protected from infection, the proportion of deaths in children under 
age 5 is small. jEvidence (figure 1) from a PAHO review of health 
coiiditions in the Americas (PAHO 1974) strongly suggests that the 
average child in Middle and Soiith America is not yet receiving 
adequate nutrition, protection from infection, or prbper medical 
care. Figure 2 shows the trends in infant mortality in Middle and 
South America, »as well as the goal$ agreed upon at Punta del Este 
(PAHO 1,974, 1971). Although the rates have declined in the past 15 
years, it is clear that excessive numbers of infants are dying in the 
first year of life, both in Middle and South America. Some indication 
of the'numbers of deaths for specific, mostly preventable, diseases, is 
given in^ figure 3. If literally thousands of children are dying 
needlessly, what can we conclude about child health services? 

As we contemplate such mortality rates, we should bear in mind a 
statement made by Dr. Cicely Williams (1953) many years ago: 

¥pw p>ersons realized that "the survival of the fittestMs a misapplied 
>cliche. It was not recognized that the same conditions that will kill 
30% of the babies in the first year of life will also produce a large 
proportion of persons with damaged lives who will be a burden for . 
years and perish at a later date. 
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Figure 2. Goals tor infant mortality reduction and rates actually achieved 
Latin America, 1960-72 [Redrawn from PAHO 1971 1974] 
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ot L.S. rates, and excess death*, by various causes. Latin America \^70 
[Redrawn from PAHO 1971] 
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Table L Neonatal Mortality by Birth Weight, Ribeiraq Preto and California 
Projects, Inter-American Invesqgation of Mortality in Cq^ldhood and United 
States 1960 Birth Cohort (deaths per 1,000 live births) 



' Birthweight 
(grams) 


1 


California 
project 




Ribeirao Preto 
project , 
1968-70 


1,000 or less 
1,001-1,500 . 
1,501-ikOOO 

2,ooi-2)i;y)o 




Wo 

434,8 
^ 106^ 
28.2 




' 890.6 
^ 696.3 
< 39V.0 . 
68.1 


2,500 or less 

2,501-3,000 
3,001-3,500 
3,50U4,000 . 
4,001 and more 




129.9 
5.4 

2.5 . 
' 2.2 
3:1 




-215.7 

8.5 
^.0 
8.7 


2,501 and more 




' 3.1 






Total 




12.7 




|> 28.2 



Souii^c: Puffer and SerrarKJ^ 1^75. Reproduced with permission of Pan American Health 
Organization ^ ^ - 

■ ' * . 

Fortunately, 30 percent of babies no longer die in the first year of 
life, but Dr. Willianr^s' admonition to remember the damage done to 
those who survive is si^rely as applicable today as it was 25 years ago. 

Low Birth Weight and Mortality 

A major cause of excessive infant mortality everywhere is low birth » 
weight. A Recent analysis showed, for example.mthat a significant 
proportion pi- racial differences in infant mortality ^tes in the tJnited 
States could be accounted for by differences in birth weights: In 
those groups witH higher mortality rates tl^ere is a higher percentage 
of babies with low birth weights (Habicht et al. 1974). Similarly, a 
recent study in. tqgland repealed that much of the variation in infant 
mortality is associated with only two factors: birtn weight and social^ 
class — and the two are obviously interrelated (Neligan 1974). « 

Variations in mortality with birth weight are striking and were 
clearly showh in the FaDi American Health Organizatiorl^study of 
mortality in childhood (Pu^er and Serrano 1971, 1975; Serrano and 
Puffer 1974). Table 1 shows the mortality in the first month of life, 
by birth weight, in two of the PAHO study sites in Brazil and the. 
United States. The high rates in the lowest weight groups are 
expected; the striking fact is that rates in infants weighing between 
2,501 and 3,000 grams, a weight range usjially Considered **normal," 
are approximately twice as high as those in infants with birth weights 
over 3,001 grams. . ' . 
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Bi rth 
we I gh t 
(gms) ^ 



■ 2500. 



2501- 
3000. 



KiKurr 4. Disn ibuiifl^oi live binhs. by l^ih wei^hi. various counuies, 1972 ^ 
[Rediaw!! from Serrano and I'uffer 1974] - - . 

■ . ''v 

The lull si^iiifi^aiue of this becomes apparent when^we examine 
other findings of the PAHO study showing the distribution of birth 
weights ill varimis places. In figure 4. it is dear that the number of 
newborns with weights that place them at higher risk is much greater 
in the Latin American sites than in ClalifoiTWa ^Serrano and Puffer 
1974). < ^ 

rhejmportance of this finding is tremendously iri*reased by recent 
observations at the Instituto de Nutricicm de Centro America y 
Panama ( INC^AP). -Stiidies there have shown that dietary supplemen- 
tation of pregnant women^can significantly increase birth weights 
(Habicfii et aL 1974); Lechtig et al. 1975a; Lechtig et al. 1975b). The 
average weight of f^abies born to mbtTiers who had received supple- 
ments providing at least 20,000 calories during either the second or 
third trimester was 200 grams higher than that.,wf babies iM)in to 
unsuppleinenled mothers. Ihiis, INCAP has demonstrated that a 
simple and not very costly iiUerventittn during pregnancy can increase 
rth weights and enhance the probability of survival. 1^ 
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Malnutrihon and Mortality ' ' . 

The association l^'tweyii inaliuilrilit)n and infection, and the fie- 
(|uently lesullii^ inortalit)i^ is well known (Scrimshaw et al. 1968). If 
we wish to understand the high mortality rates that prevail in many 
parts of the WesteriV Hemisphere, we should examine The data from 
Puffer and Serrano, whose study (1971) showed clearly the role of 
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'•Fi^uic 5. PcrccniaKC of deaths in children under age 5> ,wiih malnutrition as 
associated cause, various underlying causes, Latin American cities, 1970 
[Redrawn from Puffer and Serrano 1971] * 

malnutritidn in ptoducing infant and early childhood mortality. From 
ii-. wealth of thought-provoking data in their report, only a small 
sample can be presented here. Figure»5 shows the importance of 
malnutrition.as an associated cause of death in children under age 5, 
as observed in six cities included in theij- study sample, three in Brazil 
^nd three in Colombia. Malnutrition, was an associated cause of death 
ifi about 50 percent of all. cases in the 6ix cities. It was a consistently 
important associated cause iij c|iarrheal diseases and measles, ranging 
) from 50 percent to almost 80 percent, and was found in 30 to 45 
rcent of ^hose children tiying of respiratory infections. ^ 
~n figure 6 the percentage of deaths in children under age 5 in 
which malnutrition was found to be either an underlying or an 
as.sociated cause is shown Jby age group. Such de^hs were not too 
prevalent in the first month of^life, bUt became more so between the 
first and the sixth months. From the sixth to the eleventh mohths, 
the proportion reached almost 50 percent and thereafter remained 
high. What is* especially striking about these findings is that malnukri- 
tion increased in each succeeding year as an underlying (i-^ ' prima\) 
«(use^of dea{h throughout the ffrst 5 years of life. Clearly, malnutnl 
tre^n IS not a problem, that , children simply' pass through in earl> 
.childhood, hut (*he that^gersists ^nd, in fact, becomes even more 
important in later childhooq.' , . \^ 
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Figure 6. Percentage of deaths in children under age 5, with main mpition ^as 
underlying or associate^ cause, by age, Latin Americ^ f970 (fRedraivn 
from.PAHO 1971] ^ ^ 

^ We know ftom a number of surveys*carried out in various parts of 
the Western Hernisphere that in many countries malnutrition in 
children under . age 5 is indeed widely prevalent. Figure 7, for 
example, shows the results of surveys carried out among Central 
American countries (PAHO 197fl). NoWti^e is the overall rate 
significantly less than 50 percent aftd it is over 70 percent in three 6f 
these countries, where almost one third gf the children have moder- 
ate or^evere malnutrition. 
1^ 
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figure 7; Percent^age. of children under a^e 5 with various grades of 
^ fAHO Tg??] '''' American countries, around 1970 [RedrawnHrom 
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Evefi if we ignore the* tragic; result^ of malrtutrition as ^tflected in 
thi^ mbrtaHty data/we^must not forget the long-lasting consequences 
of maljiutritior? in these children, purely we must acknowledge *that 
nialtiutrition *is a major factor in determining the health status of 
yoyng children in many^parts of |h«w\niericas loday. This concept is 
not new^^^^diatricians, nutritionbts, and othersjiave been well aware 
of t-he problem df ^malnutrition in Latin America for the last. 15 pr. 20 
years\ DozenJ t>X nati^lfal ahd internationalVonferences have bjfen 
heldf thefeSfmds of articl<^and prob&bly scores of monographs have 
been published ■ Yet in many parts of this hemisphere the problems 
remain essentially the same. Surely, eradicating malnut^rition in* 
children ujid^K^ge 5 poses the most serious of all challenges those 
'concerned\ith the^alt'h of children in the Western Hemisphei^. . 

Breast Feeding and the HealtrTof Infants ' , 

Malnutrition in early childhood is the result of- a greaJ: many 
interacting factors. We kno\|, however, that breast feedili^g in the very 
early rnonths of life provides fully adequate nutrition, as well as 
protection i^rom infectiorts. Unfortunately, w? also know that breast 
feeding is being abandoned in many jDarts of the Americas, especially 
in the urban areas.' This trend must be a matter 'of serious cQncern to 
^1 pediatricians and others responsible for the care of 'children. The 
increased risks associiufed with artificial feeding of infants* were 

Table- 2. MorbicHty and Mortality Rates per 17)00 Infants, Ages 1 to 9 
. Months, and Faialitv per 100 cases, by DiaRnosis and Type of Feeding, 
Chicago, U.S.A.; 1924-29 v ^ ' 
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apparent 'ifi the late 19th century (Knodel and van de Walle 1967). 
Repeated studies since then have confirmed* the early findings 
(Howarth 1905, Davis 1913, Woodbury 1922, Crulee et al/ 1934, 
1935, Robinson 1951). Table 2, for example, shows the variations in 
morbidity and mortality rates by tyjJe of feeding and by selected 
diagnosis during the fir$t 9 months of life, as Ihown in a^'&tudy 
carried put in Chicago. in the late 1920s. TRere are striking differ- 
ences in mortality rates from respiratory infections, didi^rheal diseases, 
.and other infections,- depending on the type of feeding. Morbidity 
rates are, ^)y no mearw so different, although they are of course 
higher in bottle-fed babies. Thus, b^st feeding appears to provide 
modest protection from infection, b«ft a great deal of protection from 
dyipg: ' , / . 

Although comparable current irtlormation from Latin America is 
limited, it shows clearly that the excessive risk associated with bottle 
feeding is surely still present today in much of that^area. In figure 8, 
the results of a study carried out in Chile a few years ago (Plank and 
Milanesi 1973) shoiv that there is an excess of mortality among 
partially breast-fed and e>en mo^e among bottle-fed babies at ages 4 




k weeks 



3 months 



6 months 



Figure 8. Mortality rates during the first year of life in breast-fed, partially 
breast-fed, and fK)ttle-fed infants; among those surviving at .4 weeks, 3 
months, and 6 months, rural Chile, 1969-70 [Data from Plank and Milanesi 
1973] 
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Table 3. Proportions of Infants Breast- Fg^ Fi^onths or 
Population, and Among Infants Dying at^6 to I I Months^, in 4 
Areax, Around 1970 ' ^ <i 



in "the 

O '.Study 





Percent Breast- Fed 


. Ratio of 
Morudity , 
Risk for * 
Breast- Fdg. 
< 6 mos. 
2: 6 mos. 


Study Areas 


Total Infant 
Population ^ 
< 6 mos. 2: 6 mos. 


Infants Dying" 
at 6-11 Mos. 

< 6 mos, 2: 6 mos. 


£1 Salvador" 
Kingston, Jamaica^ 
MedeHin, Golombia** 
Sao-Paulo, Brazil* 


20 80 ' 
. 51 49 
61.8 31.2 
77.2 - - 22.8 


78.0' /22.0 
87,4 12.6 
91.3 ' 8.8 
95.9 4.1 


14.2:1 
. 7.1:1 
6.4:1 
6.8:1 



Sources: (a) Puffer and Serrano 1971. (b) Menchu « al. (1972). (c) Granthan*- McGregor and 
Beck. 1970. (d) Oberndorfer and Mejia ( 1968). (e) lunea et al.4975 




8' 8, 

O" 

o 6. 

s ^ 











CHILE 














11 




* fH- ■ ■ ■ ' 






^ r— 1 — W — T — 



Ag« -o -o -o g u 
group y i ^ <>j 



V 



Figure 9. Ratio of mortality rales at various ages under 5 years to Canadian 
rates for those ages, four Latin American countries, 1972 [From data in 
PAHO 1974] 
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weeks. 3 months, and *ven at 6 months. Fu'ith^r confirmation of this 
IS shown in table 3. based on mortality datk from (he PAHO study 
and from four other studies of the prevalence of breast 'feeding ir 
the total population of four Latin American cities "(Menchu et al 
972. Grarhham-McO-egor and Beck 1.970. Oberndorfer and Mejia 
1968. Junes et«l. 1975^ It is clear that the ripk of death between ages 
6 and 11 months is 7 to 14 times greater almong infants who were 
breast fed for less than 6 months. I . 

; Indirece evidence 6f the protection provided by breast feeding is 
seen in figure 9. which shows the ratio of th^ 1972 mortality rates at 
various ages during infancy and childhood' in a number of Latim 
American countries to those for the same kges in Canada XPAHCW 
1974). As Monckeberg pointed out some iears ago (1^70). where 
mortality ratios are high in the first year V life, we can expect that 
bottle feeding is common and produces e^rly. severe malnutrition. 
High mortality ratios in the second through fourth years of life are 
more indicative of a poor environment as well as ^e limited 
availability of medical care." If Monckeberg^s conclusions, are correct 
we can assume, a^^ he did. that in his comjtry. Chile, bottle feeding is 
indeed a problem while health status afid medical care are relatively 
better beyond the age of 1 year. In Venezuela, on the other hand, 
bottle feeding may be somewhat less 'common, as reflected by lower 
ratios under age 1 year, but health care after the first year of life is 
probably less adequate than it is in Chile. 

Diarrheal Diseases in the Children of Latin America 

For many years we have known that respiratory infections and 
diarrheal diseases are the leading causes of death in children under 
age 5 throughout the developing world. It is instructive to review the 
infant mortality patterns in New York City, by cause of death 
betvveen 1898 and 1931. as shown in figure 10 (McDermott 1966) 
Uuring the 33-year period infant mortality fell from 140 per 1 000 
hve births to well under 60 per 1.000 live births. As is clear in' the 
figure, much of this decline in mortality occurred because of the 
steady decrease in deaths from diarrheal diseases and from respira- 
tory infections. McDermott (1966). as well as Dubos (1966) has 
pointed out that a* no time during this period did tfee health 
professionals have available any specific effective preventive or curative * 
measures for either of these conditions that were killing so many 
infants. The number of deaths decreased, they suggest, because of a 
variety of improvements in the standard of living, particularly in 
nutrition, rather than because of any specific, easily transferred 
technology. 
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1900 1905 1910 1915 ' 1920 1925 1930 



Figure 10. Infant mortality rate, by cause; New York City, 1898-1931 
[Redrawn from McDermott 1966] 
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Figure 11. 1972 infant mortality rates and prop>ortion of deaths due to 
diarrheal diseases in Latin American countries, plotted against infant 
mortality rates and diarrheal deaths in New York City, 1915-50 [Data from 
McDermott 1966. New York City Health Department 1976, PA HO 1974] 
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The status of diarrheal diseases in the Americas today is shown in 
-figure 11. The infant morulity rate and the proportion of that rate 
due tp diarrheal diseases in New Vork Cit^ between 1915 and J950 
was drawn. Then, on the New York infant mortality curve the 
current infaqtv mortality rate for a nurnbcr of Latin American 
^countries (PAHO 1974) was plotted ayj the proportion of that 
mortality due to diarrheal diseases indicated with a bar. As the figure 
shows clearly, current infant mortality rates in these countries are in a 
range comparable to levels in the United States between 25 aAd ^0 
ytears ago. It is interesting to note, furthermore, that the proportion 
of mortality due to diarrheal disease in contemporary Latin countries 
is approximately equal to that in the United States when infant 
mortality rates were equally high. Given the importance of diarrheal 
diseases as a cause of death in infants and young children, alnd tHE 
association with environmental factors including nutrition, it is clear 
that much remains to be done. 

Health Status of Children and the Economy 

For at least 100 years, physicians have recognised a strong correla- 
tion between the socioeconomic status of families atid mortality rates- 
in infants of those families (Newman 1906). Table 4, for example, 
shows the infant mortality rate by income group in Sweden over 50 
years ago (Titmuss 1943). Note that in the highest income class, the 
infant mortality rate was already below 15 per 1,000 live births— a 
level yet to be reached in the United Sutes (Wegman 1976). j. 

Some evidence of the exquisite sensitivity of child health to 
economic changes as reflected by mortality rates is shown in figure 
12. There, we see cKanges in economic trends as indicated by 
unemployment rates, and changes in the . trends of infant death rates 
within the first day of life, between 1915 and 1965 (Brenner 1973). 
These variations in mortality in .infants in the first 24 hours obviously 

Table 4. Infant Mortality Rates, by Income Class. Stockholm, 1918-22 



Income. Swedish crowns" 



< 4.000 



4.000- 
5,999 



6,000- 
9,990 



> 10,000 




24.0 
24.9 
48.9 



15.1 
23.2 
38.3 



19.7- 

12.2 

31.9 



11.4 

2.9 
14.3 



Infant 



• U.S.$I - Sw. cr. 3.3 (±) 



Source: Rielz. died by Tumuss. 1943 
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1920 1930 19^0 1950 I960 

Figure 12. Association between deviations in infant mortality rates in the first 
24 hours and in economic trends, United States 1915-65 [Redrawn from, 
Brenner 1973] • . 



cannot be due to differences in their external environment. Thiey 
must instead be related to differences in the mother's circumstances 
or physical condition, which can be due to economically related 
environmental circumstances. The findings suggest that as economic 
conditions worsen, indicated by increasing unemployment, something 
happens in mothers that increases the likelihood^ that infants they 
produce will di^ within^ 24 hours. Given the ^well-established impor- 
tance of birth weight as a factor in infant survival and the recently- 
available evidence concerning the impact of maternal nutrition on 
birth (Habicht et al. 1974, 1975;^Lechtig, et al. 1972, 1975), it seems 
altogether likely that what we are seeing reflects the-* impact of 
economic conditions on the nutritional status of the mother — which, 
in turn; affects the birth weight and thus the survival of her infants. 

Evidence that economic factors have as p)owerful an impact on 
infant mortality in Latin America as in the .United States is presented 
in figure KS, which shows the infant mortality rate in Sao ' Paulo 
between 1950 and 1970 and j^eal income during that same rperiod 
(Leser 1973). As income improved between 1950 and 1960 there was 
a generally substantial decrease in infant mortality. As real income 
declined between 1960 and 1970, infant mortality rates began to rise 
and were finally as high in 1970 as they had been in 1950. 
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Figure 13. Variations in real income and infant mortality ratesj Sdo Paolo, 
Brazil. 1950-70 [Data from Leser r973] 



The health status of children, as npted earlier, is a powerful 
indicator of the economic status of the mass of ordinary people. The 
evidence, however, indicates clearly that economic factors "are power- 
ful determinants of the health of children. As physicians we surely 
have an obligation to apply our medical knowledge and skills to 
improve the health of children in the Americas. If we truly wish to 
help the children of Latin America, we also have an obligatior\ to do 
what wfrcan to eliminate poverty. 

Eliminating poverty me^ns more than merely increasing the gross 
national product.' Figure 14 s,hows the relation between income per 
pcrsdn and infant mortality rates in a number of countries (World 
^Bank 1975, UNESCO 1975). There is a clear general trend: As 
income goes up, infant mortality goes' down. More interesting, 
however, are the exceptions to the trencji. Some countries with high 
incomes have relatively high mortality rates; others have low incomes 
with relatively ^ow tnortality. How can we account for these excep- 
tions? We do know that in man^^jfbuntries with higher income and 
higher mortality fates.the richest 5 percent Qf the population collect 
from one. fourth to one half of the income while the poorest 40 
percent receive 15 percent of the income or less, sometimes much 
les^. The available estimates of the income shares received by the 
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Figure 14. Asaocialion between per capita income and infant mortality rates, 
various countries, 1972 [Data from World Bank 1975, UNESCO 1976] 

richest 5 percent and the poorest 40 percent in the Lflft| American 
countries included in the figure are shown in table 5 (Addman and 
Morris 197i). The situation of the very poorest is worse. In Mexico, 
for example, the poorest 20 percent of the population attempt to 
survive on 1.3 percent of the income (Barnet and MuHer 1974; see 
also Power 1975). The poor health status of many children in Latin 
America is not surprising. . j 



Table 5. 
tries 



Income Distribution Estimates in V^irious Latin American Coun- 



Percentage Share of Income 



Country 




Poorest 
40 Percent 




Richest 
5 Percent 


Brazil 




12.9 ^ 




a8.40 


Chile • 




15.0 




22.60 


Coloinb» 


7.3 




40.4 


CcMU Rka 




13.3 




35.0 


Ecuador 




16.9 




33.7 


Mexico 




10.5 




28.5 


Panama 




14.3 




34.5 


Peru 




8.8 




48.3 


Veneiuela 




13.4 




23.2 



SOUBO: Exccrpced from Adclman and Morris. Economic Growth and Social Equity in Developing 
Countrici. Sunford Univenity Press. 1973 
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r On the other hand, we know that where income is low and infant 
mortality is low as is the case in Sri Lanka and Kerala, there has 
usually been significant land or income redistribution (Gram 1976, 
Ratcliffc 1976), There is more to it than that, of course. Both Sri 
Lanka and Kerala have extensive (even though not expensive) health 
services, and literacy rates arc high, for example. But the result of 
such factors, coupled with more equitable income distribution, is a 
minimally adequate standard of living for the poorest families — and 
their children benefit. 

Conclusions 

This brief sampling of available data concerning the status of 
children in the Americas indicates very clearly that among infants 
and children in Latin America, there is still much suffering that is 
undoubtedly preventable and there are many thousands of needless 
deaths. As we have seen, many of these deaths Sfe associated with 
malnutrition; many others are caused by dj^^^ses which are com- 
pletely preventable by immunization or prompt and often simple 
treatment. The health services required to prevent these needless 
deatRs are neither complex nor terribly cQstly: a decent diet for 
mother during pregnancy and lactation; breast feeding early and an 
adequate wej^ning diet later; immunizations;' simple, early treatment 
for respiratory infections and diarrhea. 

If this sho'rt list of health service measures— ^which are surely not 
mysterious, or even sophisticated — ^^were provided, infant and early 
childhood moi*tality could easily be cut in half A decent standard of 
living for those children and their families in the poorest population 
groups would haveithc same effect. What are wejjj^iting for? ^ 
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CHILD HEALTH IN LATIN AMERICA AND THE . 
CARIBBEAN 

« • * 

Jorge V, Rosselot 

* ' iNTRpqucnoN 

The processes of growth and development which are the outstand- 
ing" chsu-acteristics of infancy and childhood can be adversely affected 
by multiple factors. Prominent among these are socioeconomic ahd 
cultural backwardness and a population growth that outstrips the 
resources of the social sectors, including the health sector. Not only 
c|^n the resultont problems affect child health and later stagcs of the 
JflRp cycle, but the well-being of the family as a whole may also be 
impaired (Rosselot 1972, PAHO 1973). 

Conventionally, the portion of the population affected covers a 
generation, which may be considered to begin immediately following 
conception and to continue through the embryonic and fetal phases 
(gestation), and then, after deliver, through infancy, childhood, 
adolescence, and youth (Rosselot 1974, WHO 1974). The cycle is then 
closed, and. a new cycle begins. ^ 



THE RIGHTS OF CHILDREN 

Children and adolescents need protection, both in the Special 
process of adaptation set in motion by development, which makes 
them especially vulnerable to the environment, and in their prepara- 
tion for a working Jife, which subsequently enables them to be 
effective participants in tiational development. 

At present, all cbuntries are well aware that the commitments 
en^jbodied in the Declaration of Human Rights must be honored and 
translated iiito an acceptable level of living, health, and well-being. 
This standard includes special care and assistance during infancy. 

Specifically, the principles regarding the protection of infants that 
were embodied in the 1959 United Nations Declaration of the Rights 
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of Children (Instituto Interamericano del Nino 1961) must be put 
into effect. These include: 

— Right to a name and nationality 

— Right to social security and health 

— Right to full and harmonious development 

—Right to family love 

— Right to education and recreation . 

— Right of handicapped children to special treattnent, education, and . 
' care 

— Preemptive right to aid ^ 

^Rigljt to protection in employment 

— Right to protection against social discrimination 

The responsibility for satisfying the needs of children falls on the 
family, the community, the State, and, to a significant degree, on 
international cooperation agencies. These needs vary< greatly. Those, 
for heailth, education, social welfare, vocational counseling, and 
employment are of high priority. Thus, fulfillment of these principles^, 
calls for a firm policy decision, which must be implemented through 
the (effective coordination (at both the technical-ajdministrative and 
intersectoral leveh) of the progi:ams of the government and private 
agencies concerned and through the continuing and active coopera- 
tion of the target communities. It is ol)vious that the indicators for 
evaluating these programs should be direct and comprehensive, 
allowing for the assessment of the various components of the health 
and welfare processes, ^ 

STATbS OF CHILD HEALTH IN LATIN AMERICA AND THE 

CARIBBEAN 

Health Problems ■ ^ 

The estimated population of Latin America in I97& was 326 
million; the projected population around the year 2000 is 606 million, 
Jhe basic demographic indicators are birth rate, 37,0 per' thousand;' 
general mortality rate, 9,0 per thousand; infant mortality rate, 75,0 
per thou^nd; and annual population increase, 2,8 percent. 

As may be seen from table 1, 59 percent of the population lived in 
urban areas,^and children uncier age 15 represented" 42 percent of 
the population. The average per capfta income was around $940, 
The table also shows that these parameters vary appreciably in the 
different subregions and countries of Latin America, 
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At present, the stage of child health in Latin America and the 
Caribbean reflects a high prevalence of preventable diseases. This 
.phenomenon is due, among; other things, to the vufnerability of 
certain age groups \o harmful environmeikal influences and to the 
current health ix)licies, • 

The status of child health is indirectly reflected in quantifiable risks 
of fmortality and morbidity during childhood/Among the indicators 
normally used to evaluate the measurement of the child health status 
is mortality in the first year of life, which in underdeveloped 
•countries is of limited validity because of the underregistration of 
deaths (Montoya 1974). These indicators do not bring out the 
^subsequent risk of mortality in the variolas stages of infancy or the 
possible health risk§ to^ survivors arising from the disabling sequelae 
of disorders that may have affected them at an earlier age, 

"Table 2 shows that in l^tin America and the Caribbean the health 
risk^sare also dften high^n childhood. This fact and the foregoing 
conjurations Vaken together refiect a manifestly bad situation, 
esMpfally when compared with that in North America: In about 
1974, deaths in chMren under age 5 in Middle America and South 
America accounted lor 37.2 percent and 31.2 percent respectively of 
all deaths, as opposed to 3.3 percent in North America (PArfO 1977). 

In Latin America and the Caribbean,'lhe leading causes of death in 
children under age 5^are enteric diseases .(around 23.5 percent), 
respiratory diseases (between 9.0 and 34J5 percent), and diseases 
preventable by immunization (apprpximately 10 percent). It is not 
easy to gauge the role played by diseases included itfi Section XV of 
the International Classification (WHO 1967, pp 291-301), "Certain 
Causes of Perinatal MprMdity and. Mortality/* whose rates, despire 
deficiencies in i^egistrationf^aye in rece^it years ranged between 6.3 
and 39.5 percefit of all deaths in children under age 5. The diseases 
mentioned are of greater importance in the neonatal period, even 
though they originate duringyhe period of fetal life. This pattern 
holds particularly true in prematurity, which was the basic or 
associated cause of 22j6 percent of deaths (Puffer and Serrano 1973). 
These statistics do not biing out the role of nutritional deficiencies as** 
the underlying caus^ of aeath in infants, although the Inter- American 
Investigation of Mortality in Childhood (Puffer and Serrano 1975, 
pp. 53-63) showed that it was £^ basic or associated cause of around^ 
34.1 percent of deaths in clildren under age 5. ' 

Of course, the; health risks of infants and children are significantly 
influenced by the health status of their mothers, especially during 
fetal development and the neonatal period. During the reproductive 
cycle, there are still serious risks (most of which are preventable) of 
dying. At the beginning of the decade they were reflected in maternal 
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Table 1. Selected Demographic and Economic Data in the Americas (Conl.) • 



Regior/orCountiy 



Tropical South America 
Bolivia ( 
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Population Raw. of 
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: World Miion Dau She«. hi.\m Reference Bureau Inc., Washington. D.C. 1976 
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Tabic 2. Death Rates Per 1,000 Population Under Age 19, by 5-Year Age 
Groups in 10 Latin American Countries 



Age, in years 



Country 



Year 


Total 


0-1* 


1-4 


5-9 


10-15 


15-19 


1969 


9.^ 


61.1 


2.8 


0.7 


0.7 


1.2 
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1.0 


1.5 
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0.8 
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63.3 


7.9 


1.8 


1.1 


1.7 
1.1 


197! 


6.6 


49.8 


5.3 


1.0 


0.7 



Argentina 
Colombia 
CoBta Rica 
Cuba 
Chile 

United States 

Guatemala 

Jamaica 

Mexico 

Venezuela 



•Per 1000 live births. ' 
SouKCX: Pan American Health Organiiaiion 1974. IJcpartmcni of Hcahh Siaiisiks. 

death rates of 13.5 per thousand and 18.80 per thousand in Middle 
and South America respectively and of only 2.5 per thousand in 
North America. Furthermore, low birth weight, one of the most 
accurate predictive indicators of child health, is notoriously prevalent 
in most of the countries of the region (Puffer and Serrano 1975a, b) 

The risks of mortality after childhood and during adolescence are 
considerably less; during this period, the leading causes of death are 
accidents, suicide, homicide, certain types of infectious diseases, 
metabolic disorders, endocrinal diseases, and tumors. During this 
period, psychosocial disorders play a preponderant role and contrib- 
ute to and/or are the consequence of, family^i^ruption. They take 
the form of drug addiction, smoking, drinking, sensory disorders 
learning difficulties, and behavioral disorders that may lead to 
prosUtution and juvenile delinquency. Especially worthy of mention 
during the period of adolescence is the importance of diseases 
connected with the onset of the renroductive cycle. These diseases 
occur as a consequence of the gradumrhange in roles and values and 
the lack of proper sex and family life education. This change is 
conducive to sexually transmitted diseases, undesired pregnancies 
and illicit abortions (WHO 1976a, b, PAHO 1975, RosseJbt 1977). 

The health problems of children and adolescents are closely related 
w^changes of various kinds in the process of overall development. 
These vary from country to country and eventually produce impor- 
tant sociocukural changes in the community, especially in families. 
This pattern is particularly true of families exposed to major 
biological and/or socioeconomic risks, especially those effected by 
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temporary or permanent migration from the rura4 areas to the cities, 
fn these cases, the protective role of the traditional extended family ' 
is minimized, especially during the traditional phase (if adaptation — a 
phase which usually takes place in the adverse environmental and 
acculturation conditions characteristic of shantytowns (Wray 1975). 

Ongoing^ Child Health Programs 

' The health protection of children and adolescents must be part of 
general economic and social development plans and family health 
and weWare programs, which must be, designed to enlist their 
(;pnscious coopelf-ation and thereby provide an impetus to the health 
•services., This was precisely the purpose of the health goals embodied 
in the Charter of Punta del Este and approved by the^first Sjjecial 
Meeting of Ministers of Health of the Americas (Washington, D.C., 
1963). One of those goals was a 50 percent reduction in mortality in 
children under age 5 in the 10-year period 1961-71. At the same 
time, it was also postulated that, to achieve this goal, the Ten-Year 
Health Plan should become part of the general development plans of . 
the countries involved (PAHO 1964). 

In the past decade, because of the political and socioeconomic 
situation in Latin America and the Caribbean, limited progress has 
been made in implementing these plans and programs, bringing 
about structural changes, accelerating economic growth, and expand- 
ing social services, including health services. This limited progress 
explains why the goals agreed upon have not been fully achieved. An 
.analysis of the health sector, and in particular of child protection, 
shows that, although the risk, of dying at an early age has been 
reduced^ it still does, not meet the target set (table 3) (PAHO 1971). 

At the beginning of the present decade, the coverage of child 
health programs in most* Latin American and' Caribbean countries 
was still limited. Health activities were carried out without the 
necessary degree of coordination with the activities of related pro- 
grams — such as medical and dental care, epidemiology, nutrition, 
mental health, health education, ^a^d community development— for 
which the Ministries of Health and other institutions of the health 
sector and other sectors were responsible. 

The scope of maternal health programs; so closely connected with 
child health, has been limited. Thus in Latin America and the 
Caribbean the coverage of prenatal care, of care during labor and 
delivery, and of postpartum care by qualified personnel has averaged 
vSO, 50, and 5 percent respectively. These types of care irrfluence the 
biological quality of the fetus and the newbori^ child (high incidence 
of congenital anomalies and low birth weight) and has an important 
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Tabic 3. Rcducuon in the.I>cath Rat^s of Childrcri under Age 5. in Selected 
Countries. Compared ji^ith the Goals of the CJftarter of Punta del Este 1969 
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Adapted from PA HQ 1971. 



bearing on health in the later stages of childhood. In addition, the 
sh(?rttomings in the above-mentioned programs reduce opportunities 
tor^nfroyidrng proper training.in basic mothering and counseling for 
tam^y^fe. includmg birth control (PAHO 1973, WHO 1976). 

■ pare of children under age 5 i's belated a^nd limited, and 
sometimes the preventive aspects are neglected. An even worse 
situation IS to be .noted in adolescence, when health care is precarious 
(%ss*lot 197?, PAHO 1973, Rosselot 1974). 

jFurthermQi-e, si>ecific family Welfare programs that have been 
tQrmulkted and are conducted in close coordination with other 

. inst.tuoons ahd other sectors are rare, even though children can 
obtam umlenjable health benefits from them. All this applies to the 
shantytowrts dnd especially to the rural areas, where health probleiiis 
parocularly 'hji children, are very serious. At their meeting in Puntk 
del Este in 1967; the presidents of the Americas recommended that 
maternal and bhild protection and family education programs be 
increased. In addition, participants at the Second Special Meeting of 
Mmisters of Health of the Americas (PAHO 1968) (Buenos Aires 
1968). when reviewing the results obtained in implementing the Teri- 
Year Health Plan mentioned above, made specific recommendations 
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Tablie 4, Evaluation of the lO-Ye^r Health Plan, 1971^80, Child (/era] th and 
Family Welfare * 



Mortality ifi Children Ages 
Infant Mortality ^ ' 1-4 

Deaths in infants under age 1 Deaths in -children ages 1-4 



Country . 


per 1 »000 live births 


per ItpOO population ' 


1971 - 


1980 ' 


197^1 ; 


1980 


Brazil 
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60,0^♦ 


^ 30.0 
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12.0 


Honduras 


117.6 


85.0 
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7:4 


4.4 
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42.4 
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Sotmcz: Adapted from PAHO 1976 
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in tlie area of child health that ^ere ainied^at expanding ^d 
integrating the progranns, particularly in the undetserved areas^ and 
at strengthening th^ education, and training of necessary personnel. 

More recendy, the Third Special Meeting of Ministiers of Health of 
the Americas (iSantiag© de Chile. 1972), which approved the Ten- 
Year Health Plan for the decade 1971-80, subsequently ratified by 
tlie governing bodies of the Pan American Health Organization, 
reviewed the most serious child health problems, set goajs in reducing 
the risks and in expanding coverage, and outlined a strategy for 
achieving the abov«-mentioned purposes and Syercoming the techni- 
cal and administrative problems impeding thf smooth implementa- 
tion of the prograims. Specifically, the documeWt recommended a 
reducdon of 40 percent on the average in maternal and infant 
mortality rates and of 60 percent In the mortality of children jn the 
1-4 age group. ' , \ 

In the area of extension of program coverage, it recommended 90 
percent coverage for health activities dealing with mothers and 
children un^er age 1 ^nd 50 percent coverage for those in the 1-4 
age group by the end of the decade. These goals are^of course 
l*pgional estimates, and the countries mu5t adjust them to local 
conditions. The goal^ estabhshcd for selected countries in the region 
are shown in table 4 (PAHO 1968). / / 

The goals of the suggested strategy and that mentioned ekrlier 
(PAHO 1973) are basically cJ^ same, afnd the^ degree of achievement 
has varied from country to 'country. Amon^ these activities, the 
following warrant mention: ■ - ^ ■ '/ ■ 



32 EVALUATION OF CHILD HEALTH SERVICES 



— The formulation of an intersectoral policy for the' protectioh of 
families, mothers, and children that guarantees their civic, legal, 
and employment protection rights and promotes recreational, 
educational, and vocational guidance activities for children and 
adolescents. 

—The preparation of a program for the integration of child health 
activities into the general health services. This prograip will cover 
all children, will be accessible to all, and will be efficient. Its 
operations will encompass the various stages of children's lives and 
include coordination with health activities to be carried out during 
the whole life cycle. In this regard, the processes of human 
reproduction will receive special attention, since they influence 
growth and development and, consequently, the health of children. 
Where appropriate, criteria will be established fon the preferential 
and early treatment of cases that merit such trea/tment because of 
their biomedical-social risk. To accomplish thi>aask, a system of 
regionalized health services which will provi^te progressively more 
complex care at each level must be established. In addition, 
activities must -be c^frrijed out by multidisciplinary teams whose 
^ efforts are directed at different^sectors of the population and 
whose members incTH^e (especially in rural areas) health assistants, 
lay midwives, and community leaders. 
—The establishment of a system for the education and training of all 
personnel^involved in the program. The system will be based on 
methods that ensure continuing educatipri and/or training geared 
to the problems and needs of each community. This system in turn 
will foster research for improving tjij^f^iency of the service (table, 

5). ; 

—The involvement of the communky-and of the fa^nilies themselves 
rn improving health conditions, bearing in mind the political, 
administrative, social, cultural, and anthropological characteristics 
of each community and 'the coordination oPthat process with 
intersecto'ral programs that 'have a significant health impact. 

—Xhe establishment of a system for the, continuous and ongoing 
evaluation of the extent to which technical cooperj^tion objectives 
are achieved. The system will include the study of indicators for 
evaluatingljhe effectiveness and the cost-benefit ratio of the 
activities Concerned and for making the necessary adjustments in 
the process where appropriate. . 

In the last 5 years, some Latin American and Caribbean countries 
ah^; have reasonably satisfactory health structures have made prog- 
i^ss in reducing the risks of mortality in* infancy and childhood. 
Th<?.re are many .reasV>n,s for this progress, but it is primarily the 
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Table 5. Priorities of Maternal and Child Protection Programs According to 
the Level of Socioeconomic Development* . 



Health 
Determinants 




Socioeconomic Level 




High 


Medium 


Low 


Prevalent Risks 




High maternal and 




Maternal health 


Low maternal 


Very high maternal 




mortality 


neo> and postnatal 


mortality and child 






mortality 


mortality 


Infant health 


; Perinatal mortal^ 








ity declining 


* ,• 




Pnonty Groups 








Maternal 


Primiparas, . ado- 


Primiparas and 


All pregnant 


programs 


lescents, unmar/t' 


abortion repeaters ** 


' - women 




ried women 






Infant programs 


Newborn chil- 


Newborn, un- 


Wy children 




dren with low 


weaned infants 






birth weight 






Priority Activities 








Family life 


Moderate 


High 


High 


education 








Primary • 


Moderate 


High 


High 


prevention 








Medical care 


Moderate 


High 


High 


Mental health 


High 


Moderate 


Limited'. 


Social welfare 


Limited >- 


Moderate 


High*^ " 


Professional Personnel 






Available 






i * 


Specialized . 


Obstetrician-pe* 


Obstetrician-pedia-^ 


Obstetrician-pedia- 




diatrician 


trician 


trician 




r^Jursey and/or 


Nurse and/or mid- . 


Nurse and/or mid- 


Basi^ 


midwife (♦*♦) 


wife (♦♦) 


wife (♦) 




General practi- 


General practi- 


General practi- 




tioner and nurse 


tioner and nurse 


tioner and nurse 




(♦•♦) 


(**) 


(♦) 




Auxiliary (♦) 


Auxiliary (♦♦) 


AuT^Hiary (***) 


Nonprofessional 


* . 

Volunteers (♦) ' 


Health promoters 


Health promoters 


Personnel Available 




and untrained 


and untrained 






community workers 


community workers 






(♦*) 


(♦♦♦) 



I ncrcaic. in numbers ^ 
Soukcb: Roueioi 1972. Reproduced with permission pf Pan American Health Organization. 
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result of an expansion in medical care coverage aiid in community 
participation, especially in the rural areas. \ 

There is a good possibility that improvements ih the statistical 
systems have obscured some of the possibly favorabiA changes that 
might have occurred in certain countries' as regards bisic indicators 
of child health, especially of infants. However, there is rm doubt that 
the international financial situation and its effects on, national eco- 
nomic and sotial development have made it difficult for most 
countries to obtain sufficient resources for the health\sct:tor in 
general and for child health programs in particular^ Sinte future 
financial developments are difficult to predict and are not to be 
influenced greatly by the health sector, a 4nore equitable\ectoral 
distribution of the national income should be encouraged whenever 
feasible: » \ 

The health sector must promote policy decisions designed to 
improve the mechanisms for inter- and intrainstitutional coordi'ndlion 
and to find rational solutions to the problems of health serV\ice 
delivery. Solutions are needed that will permit an increase in the 
operational efficiency of the programs, while keeping the coit-beneflx 
ratio of the tasks be undertaken (PAHO 1968) at an, appropriate^ 
level. ^ \ 

There is no doubt that a:t present much is to be expected from the/ 
innovative medical care approach in which child Jjo^alth problems are 
first analyzed and their control is then programed in a broader 
context that extends beyond traditional maternal ^arjdi child health 
care to the promotion of the health of the entire family including the 
biologically most vulnerable family members (Rosselot 1974). It is 
promising that the concept of family .tj^palth Is not only being 
progressively accepted in Latin America- and the Caribbean, but is 
also being gradually incorporated into the operations of the general 
health services. It appears to be increasingly better understood in the 
region that, if families are to perform their full biological, psycholog- 
ical, social, /rultural, and economic functions, more satisfactory condi- 
tions for the promotion of child health musfbe established. This calls 
for the institutionalization o^ the above-mentioned process, based 
prin^ily on the motivation of the community (NewelN4g 7 5 ) and on 
the proper training and supervision pf the health team resffensible 
for the activities concerned (Flauhault 1970). 

In the rural areas, the promotion of family health must be the 
focal point of the primary health care system.Mt has been taken into 
account in the Sixth Program of Work of the World Health 
Organization for the period 1978-83 (WHO 1976a). The new 
strategy, which is strongly supported by the United Nations Chil- 
dren's Fund, calls for the "expansion of the basic services f6r children 
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in developing I countries** (United Nations 1976)/ It- is a formula 
which* given r^lute community participation, will gradually put an 
end to the present underdevelopinent in the social and health fields. 
When this task is accomplished, childfen will undoubtedly be the 
principal beneficiaries. 

With the technical cooperation of the Pan American Health 
Organization (1976), the Latin American and. Caribbeah countries are 
making headway in implementing that policy. It b the only feasible 
alternative for .sjecuring, within a reasonable period of time, a more 
adequate level of health and well-being for the underprivileged and 
needy communities and for their most valuable and potential human 
resource, the children of the Americas. 
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^ QUESTIONS BEHIND THE ANSWERS: WHY WE DO 
NOT EVALUATE AND WHY WE SHOULD 

Abraham Htmoitz 

The fact that mortality in children under 5 years of age, along with 
the constellation of causes that bear on it, is the most significant 
health problem in the developing Americas has been brought Out by 
the presentations of Dr. Wray and Dr. Rosselot. These children are 
historically poor and they live in physiological misery — which means 
that they are continually at risk of disease and interference tn their 
physical and intellectual growth. They are the victims of underdevef- 
opment. Medical technology, essential as it is, js only one factor in the 
organized efforts to combat the deleterious effects of history, the 
environment, lifestyle, a cumulative genetic heritage, prevailing pol- 
icy, and the economy. 

The progress in this underwriting over the last 30 years is evident, 
but we have not evaluated it systematically, with an overall view of 
the problem. The volume of the investment has not been small, but it 
has always fallen short of the tangible neled. We would have 
preferred, in the same period, fewer arms and more food, less costly 
health equiphnent and more proteins and vaccines. Still, we have 
failed to measure maternat and child health programs rationally in 
terms of their structure, functions, and effects. At best, we have done 
no more than register specific morbidity and mortality rate^, and 
even this effort has been incomplete, carried out on the basis of data 
whose precision left something to be desired, as did the timeliness 
with which they were examined and disseminated. In general we have 
done it after it is too late to modify our course of action. 

We know perfectly well that statistics has an intrinsic value as an 
analytical language which describes phenomena or situations and, on 
the basis of experience, helps us to predict the way they are likely to 
evolve. It gives a picture of things as they are and helps to focus 
them more closely. It is essential for shedding light on the past and 
forecasting the future. Furthermore, as events take shape, statistical 
data are irreplaceable as a means of adjusting the decisionmaking 
process. In short, they are the b»asis for modern systems of evaluation 
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and information, which are part of the process of economic and 
social development planning. There/ can be no viable programming 
without reliable statistics. Some 25 years ago, Frost (1951) wrote it 
this way: 



. . . for while the facts expressed in a statistical record constitute only 
a part of the evidence required, they consdtute the part which can 
be most conveniently and forcibly presented and are essential to any 
quantitative statement of results achieved. 

If we really desire to submit our judgment of what is being 
accomplished to thS^^ ripal test of st^dstical evaluation we must set 
about collecuhg stadstics which will serve the purpose. 

The fact is that we do not evaluate on a regular basis; if we do, it is 
exceptional— as in connection with ^search that has clearly circum- 
scribed objectives or with categorical pt'oblems, such as the control of 
a communicable disease. For the most part we fail to evaluate health 
programs and general services — andi by extension, complex social 
undertakings — as an overall system. And when we do, it is less often 
than we would like that our recommendations lead to their desired 
specific effects: namely, the creation of laws, standards, or proce- 
dures; the changing of a policy; improvement in the design of a 
program, the definition of objectives^ or the planning of investments; 
or fulfillment, generally speaking, of the true purposes of evaluation 
in terms of giving priority to problems, reducing their magnitude, 
and increasing the efficiency and effectiveness of programs (Goldstein 
1973, pp. 205-206). Flagle (1973) has said: "Information is potentially 
valuable only to the extent that it influences decisions and actions. It 
is actually valuable only to the extent that it influences outcomes 
favorably." And he adds: "The value of information gained from the 
evaluation process is ultimately the value of change in programs in 
response to evaluation. If we are speaking of large-scale progr^rii^, 
we must add to the objective measures of improvement the sul^lr4tive, 
ones assotiated with basing policy on knowledge rathei^- tihin^on , 
pressure, hope, or delusion. Also, we should weight the | electee' 
value of acting on the evaluation reports by the probability that^ueh* 
response will take place." ' ^ \^ 

It is appropriate to ask why we do not practice evaluation \n ttxe. 
developing countries. The answers to this question represent perioriai 
value judgments and generalizations in the complex world ofiheaJth 
care that might be open to criticism or unfavorable commeht.; 
[Nevertheless, the fact alone of enunciating them is reason enough fpr . 
holding this conference, whose tinxeliness it is important to empha- 
size. ^' 
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We do not evaluate because the basic information and the program- 
ing ijjiat results therefrom are deficient. The definition of targets is 
not always precise, so that the periodic measurement of their 
attainment tends to lead to rather subjective interpretations. Kane, 
Henson, and Deniston (1974, pp, 217, 230) maintain that objectives 
are rarely explicitly stated and often lack conceptual clarity. 

Health planning as' a differentiated discipline is relatively new in 
tlie AiAericas. It has come into its own in the last 15 years as an 
instrument for irrdiicing social progress, not as the expression of a 
given political structure in which decisionmaking is totally centralized. 

Planning has been officially accepted by the governments. How- 
e^ver, it is not done regularly in actual practice. Not all the countries' 
have formulated a natioil^ health policy or identified their priority 
problems, settling goals for each that are consistent with the available 
resources. Moreover^ their decisions have laAed continuity. 

What we have said can be summed up in the statement that 
evaluation is not possible without viable programing, and viable 
programing, in turn, clepends on having measurable targets. This 
explains — although it does not justify — the fact that systematic evalu- 
ation to ensure the quality of health care is practiced only as an 
exception. Pamela Horst and coworkers (1974) have pointed ^ut: 

Many federal programs are not evaluable: no measurable objectives 
are officially established; intended uses of evaluation studies have 
not been clarified by program managers; agencies are under 
pressure to spend substantial funds for evaluation yet lack the 
leverage for applying meaningful criteria to them; and the means of 
controlling evaluation funding are both ineffective and inefficient. 

• • * -^^ 

SchiT)idt et aft Xl;S(75).'j^ their part, that "while evaluation 
af fike^S de fine -dritena they 'fa^il }to specify adequate m ea su res of 
v.-iuiGcess."':; , - ' >. ; 

We fait Hp Evaluate interna ticaU^^ J^^^^ because the 

"state of tlhe^art^Vis rudiilhietLt^ry (NaU^^i^a^emy of Sciences 1974, 
p. 11) — which is th^ sanie^as.sa||j[ng tU^^<.^fj[^^^^^ consistency yet in 
the prin<:iples and meth^s tnl^i.t|^<^^^ 

cortseh$i|s ;bas ^ reachi^d Tri^gs^^ data that should 

be obtaitijfdP^'and analyzed*Jh[*^^ health care. Lewis 

(1^74>'*tc?ininds us that in tK:e^:)[5j|i^^ assessment of the 

quality of care has cotne full cif^e^S^^p^^ ago it was the final 

results that were lliie principal, ^jUj^ evalu^ion; subsequently^ 
their place was taken by the resoiirces and their characteristics, and 
by the processes. Today, once again, the results are evaluated, but 
bearing in mind that they depend on the methods used. 
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Some experts believe that information, which reflects structure 
(organization and resources) and process (health actions), is more 
important tfian the outcome (Donabedian 1966, Morehead 1974, pp. 
53, 54). Others maintain that the quality of services should be 
measured inij^iicrms of the final results — that is to say, the recovery of 
health by the sick and their active reintegration into society (Roemer 
1971, Williamson 1971, National Academy of Sciences 1974). Both 
approaches have, as it is to be expected, serious limitations, which 
have been analyzed by Brook (1974, pp, 183-210). From this, 
difference of opinion there arises the -urgent need — which can be 
expressed in another series of questionsr— to investigate the real 
correlations between data on structure and process, as well as between 
these two and outcome. It is recognized that what we know about 
structures and processes should be interrelated, but the precise 
nature of the relationship eludes us. This problem has led Barbara 
Siarfield (1973) to propose an integrated research model and to point 
out that: 

Public pressure, exerted throuj^h legislation and administrative 
agencic^vv.will increasingly require that the results of medical care be 
demo^striiic<l ... If structural and process criteria ^re indeed to 
coiuinq^^^^V'be useful indicators of the "quality" of medical care, 
systematic efforts to relate them to outcome should be initiated 
without further delay. 

This, in my view, is the most important task if we are to move 
toward a rational approach for the evaluation of health systems. It 
lies^at the very center of the interface between research and medical 
practice. 

In the meantime, I think that for the developing world it is useful' 
to measure, both in institutions and in communities, the resources 
invested and the activities performed toward achieving given objec- 
tives for prevention and cure. As we record our accomplishments, we 
should point out that they are not the exclusive consequence of 
health technology; without it, however, results would probably be less 
tangible in many cases. 

We would like to cite^ recognizing their intrinsic weaknesses, the 
three evaluations of the Ten-Year Health Plan contained in the 
Charter of Punta del Este (OAS 1961; PAHO 1966, 1968, 1971). 
They include measurements, for one region of the world, of steps 
taken and results achieved in regard to the goals for the respective 
priority problems agreed upon by all the governments. The fact 
alone that they served as an incentive and as a model for the 
formulation of the health plan for the current decade is worth 
pointing out. We are confident that this exercise will continue— each 
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time on an increasingly solid basis — with objectives set in terms 5f 
resources, activities, and results, their regular evaluation being an 
integral part of the health system in each country. 

We do not evaluate, it is claimed, because we lack definitions, 
indicators, and standard classification schemes at the national and 
international levels (WHO 1971). The methods that we now use are 
not always appropriate? for measuring actipn and outcome. This, 
then, is another area of investigation in which a series of questions 
cart be posed. 

It might well be asked why, despite the logic of the concept, 
evaluation is practiced indhe health field more as an exception to the\ 
rule and on an experimental basis than as a regular activity. In m% 
view, those who decide on and carry out programsjeact in a singular 
way to assessment Of the expected results, The^ seem to have a 
psychological inclination to content themselves with whatever effort is 
made, rather than to be concerned with its immediate and long-range 
effects-^that is, the degree and characteristics of the chang<&s induced. 
It is a subconscious evasion of the responsibilities inherent in their 
work. If the real accomplishments do not coincide with what has been 
projected, the magnitude of the failure is commensurate with the 
rank of the official concerned, the reaction on the part of societv and 
consumers, and the size of the investment. There are those wh3^ will 
say that this interpretation is baseless^ — and they may be correct. The 
fact remains, however, that the best planning models habitually fail to 
be explicit in regard to evaluation. At the same time, it is not hard to 
show that there are programs of questionable value — be it because 
they are unproductive, untimely, or unnecessary — which tend more 
to serve political, cultural, social, or administrative interests than the 
human beings they are designed in principle to benefit. The absence 
within the program design of clear-cut objectives and guidfelines for 
measuring their attainment obviates the neec) to qualify their useful- 
ness and thus allows them to continue, despite their low apparent 
yields 

Patton and coworkers. ( 1 976) have conducted an in-depth analysis 
of 20 evaluation studies in the health field, taken from an overall 
sample of 170 studies. They conclude that the recommendations are 
followed in direct proportion to what they call the ^'personal factor": 
in other words, the manifest interest of^hose who are responsible for 
health decisions. Evaluation serves to reduce their uncertainty in the 
face of the complexities of social reality and to support their courses 
of action — not necessarily to exercise an impact on national policy or 
on the broad lines of programing. Thus, decisions are usually of a 
political nature and not ?ilway* based exclusively on scientific tenets. 

We lack the necessary evaluators. Furthermore, we do not have a 
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^lear understanding oiF whether evaluation is a separate discipline — a 
specialization within the* health sciences— or only an attitude, a way of 
thinking and acting on the part of professionals, whose training, in 
any case, should be based on the principles of modern epidemiology 
(White and Henderson 1976). ' 

Developing countries also lack simple and reliable methods for 
organizing evaluation as a continuing, . rather than an incidental, 
process. This problem can be traced to the demands on the part of 
certain sources of foreign capital for cost-benefit and cost-effect 
studies in order to set priorities in health — techniques whose practical 
applicability to the social sectors of the economy has yet to be 
demonstrated. 

Despite our insufficient experience and the series of questions that 
are still awaiting answers from research, we must'evaluate. It is in our 
interest and it is our moral obligation to do so. The State, in order to 
protect society and guard its investments, is beginning to insist on 
concrete results. An expAiment, unique in the world, to ensure the 
quality of medical care hay already received legislative backing in the 
United States. Many governmejits are concerned about providing 
access to services for all their people, but none of them as yet has 
established a national system that guarantees the quality of such 
benefits. We recognize, of course, that it is a long-term process; its 
immediate goals should be liniit^d at first, and its guiding procedures 
should be carefully established (National Academy of Sciences 1974, 
Haggerty 1974, Cambridge Research Institute 1976, pp. 131-136). 
SuD^ly other countries will adopt legislation of this kind in the future. . 

Evaluation is imperative, however. The growing competition within 
each government for the distribution of national income in relation 
to priority problems makes it rtecessary. Underdevelopment is char- 
acterized by a marked imbalance between vital needs and the 
resources that are available. At the same time*, the' social sectors are at 
a great disadvantage vis-a-vis those that are considered "directly 
productive.' .We have not even learned, let alone used, the language 
of th^ecohomists to defend our proposals. We do not want to believe 
t,^*f^ arguments in terms of the meaning of life— not 

ri^^satily its quality — and of disease and death fail to impress those 
who have the power to decide what order of investments in health 
and in food are compatible with a civilized society. 

If we do not evaluate, we will not be able to explain to the State 
and to the communities what the social effect of our efforts, and 
their money, has been. Nor wj^^l we bie able to ask for increases, 
because we will be unable to justify the results of the preceding 
investment. .,< 

If we do not evaluate, we will not be able.io identify the defects in 
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the organization of preventive and curative serviceA^or the waste, 
the duplication of effort, the unnecessary examinat^on^S^nd interven- 
tions, the iatrogenic diseases, or. in sum. the ineffective administra- 
tion of institutional, ambulatory, and community care. It is claimed 
that in the United States the overall waste in the annual health 
budget of more than-$130 billion is no less than 25 per cent (White 
1976, p. 71). A similar proportion is estimated for the developing 
countriies in the Americas, but there are many added problems as 
well: the inadequate utilization of scarce human resources— the 
example^ of nurses being a conspicuous case in point (Secretaria de 
Salud Publica.-Asistencia Social. Republica Dominicana. 1974. Grupo 
de Trabajo de Enfermeria. 1973); an enormous expenditure on the 
construction of hospitals in which the quality bf care is sacrificed for 
sumptuous, and not necessarily functional, design; an investment in 
complex equipment which we tend to purchase in excess of our real 
needs. If these funds were spent on maternal and child health, 
nutrition, and systematic immunization, they would generate greater 
health and higher levels of economic production. In any case, if the 
cost of medical care were reduced without a decline in quality, many 
more people could be treated. \ 

If we do not evaluate, we will not be able to iny>rove the planning 
process and periodically readjust our objectives in terms of the" 
human, material, and financial resources available. Nor will we be 
able to guide the decisionmaking process in line with the purposes of 
the health function. 

If we do not 'evaluate, we will not be able to argue intelligently for 
reforms in health sciences education. For we want the teaching 
process to give greater priority to the prevention and not just the 
cure of disease. To this end. we want it to become more community 
oriented and not to remain institutionally based. At the same time, 
we want the learning process to result in an. understanding of the 
major problems and their feasible solutions, and not merely in an 
analysis of disciplines that reflect the program structure in the schools 
of .health sciences. We want the physician who emerges from this 
educational experience to have a view of human beings as biological 
and social entities, and to have a genuine interest in the society that 
he is going to serve. 

If we do not evaluate, we will not find out what still needs to be 
investigated in order to more appropriately measure resources, 
functions, and effects in relation to the objectives. It will be more 
difficult for us to adapt nlethods that are applied in other societies, as 
it will be to identify those professionals whose effectiveness is less 
than what is expectfed in order to motivate and educate them. 

If we do not evaluate, we will not be able to adequately meet our 
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moral responsibility to contribute ^p- social well-being and to the 
common good. 

In summary, the advantages of evaluation are evident, although 
the difficulties are still considerable. Our questions "Why don't we 
evaluate?" and "Why should we evaluate?" suggest a series of other 
questions, in turn, in regard to this poorly defined area which reflects 
the sequence between science and practice. We do not yet have 
universal indicators and methods for evaluating general and specific 
health programs— a responsibility of research/ But we do not want 
procedures that are excessively complicated, for they will inhibit the 
administrator who is responsible to the political powers and cause 
him to lose interest. In this sense, I hope that this conference will 
make some useful contributions. 
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PRAGMATIC 4^SPECTS OF PROGRAM EVALUATION 
REGARDING CHILD CARE AND ITS RELATION TO 
DECISIONMAKING AT THE POLITICAL- 
ADMINISTRATIVE LEVEL 



The Need.for Evaluation r ^ . ■ 

One third of itiy professional life has been cjevotcd to administra- 
tion and two thirds to the pursuits of reseai^ch, teaching, and 
professional practice. With .this in mind, it js obvious t^at the "biases 
introduced by these diverse activities will affect my conceptualization 
of an issue. Since I share Gunnar Myrdal'^ viewpoint (1967) that no 
one can completely eliminate his biases, I have decided to follow his ' 
advice and frankly declare mine at this point." 

Undoubtedly, the establishment of priorities is the main focus of 
an administrative effort, especially in developing countries,.where the 
gap between resources and problems is more evident and where the 
magnitude, quantity, and severity of those problems are equally 
apparent— especially as related to children; who constitute a high 
percentage of the population. '. \ * 

In dealing with this pro'Blem, the first question that could be asked ' 
is the following: Can ari entire program be evaluated? In genf^ral. the 
immediate response would be affirmative; nevertheless, an absolute* 
answer would probably be erroneous from a practicai.'standpoint. We 
could try to answer the question , after analyzing a few concrete 
experiences. * 

A pertinent exahfiple is' the history of endemic goiter Hi Colombia. 
About 150 years ago, M. Boussingault (1849). who had. come to 
Nueva Granada to study other problems, observed tJ^je correlation 
between the presence of goijer and the lack of iodijpi in food, 
especially in salt. He also noticed the absence of goff# in places- 
where salt containedj^sufficient iodine. BoussingaiA never quantified 
this phenomenon, because it was so obvious that such action seemed 
unnecessary. The governments existing in those di^yi ignored his 




V 



48 EVALl^ATION OF CMILD HEALTH SERVICES*' • ' 

rccomtiifendationPin spite of his clear and %nvincing. argument thaii 
iodizing salt ' would solve the •problenri. Accordfhg' to the i*petite 
Wstoire" the timqn> one ruler, who i^nderstood* an8 believed 
lJoussinJfault*$ .worct refusecb^even t<j. consider thf possibility •f 
^ * iodizing salt because **it wai easier to,irule*those yMo had gpiij8ir."*This 
# example ^ows'iftat so^|netimes approfyiate decisioni* are not n^e, 

* . either because of rcasopshhat have no relation to the 'problem ifillf. 
r ^ or because an jpdminiStratpr has ^j^iTferent percepti(^ of the 

: • problem.,, ' ^ ' ♦ > 

• An evaluation conducted *some^30 y^ars ago (Parra^ 1948) pointed^ 
out. that^more than half of the school population iy Colombia' 
' suffered from endemic goiter. It -is "highly^ probtible that in out 
^ coUTitries, given the socioeconomic structure qf the population. tTO- 
nonschool population^at the time suffered fronrendem*ic goiter at a 
similar, or even higher, rate. A comparison of the costs of ibdiling 
salt and those of treating goiter ind its tonsequence|*was a determin- 
ing factor in the positive action taken to solve t\\is problem. In this 
case;, the administrator used the results of the evaluation as'a basis 
for making a political decision; information resulting from a scientific 
study was Utilised. The two examples are not strictly comparable, 
because other factors.pertaining to the time, values, and technological 
advances were not the same. Still later, researchers wanted to perform 
an evaluation 6{ the results of .^alt iodizatioa, but by* that time, the 
results were evident: Endemic goitef .as a public health prgblem had 
disappeared. ^ 

It is worth mentioning that sometimes a researcher confuses the 
means and the ends^ either because of ap excess of aciidemic zeal, or 
simply because funds are available foj- research. Then the researcher 
may commit himself to an . investigation without considering it* 
relevance and usefulness for the country as a whole. Fortunately, the 
abovertiefiitioned project did not take place, arict resources were* 
channeled irKo the study of other problems of Higher priority, such . 
as malnutrition and the causes of high irifar>t(itiortality rates. - 

As shown in the above examples, factors contributing to a decision 
on the practicality of evaluating a program are multiple and should 
be considerfrd bfefore any commitment is made. We should look at 
' ^the availability of resources, the practical utility of the resulft for the * 
de4:isionmkking process, the usefulness of the study, and the oppor- 
^ tttnity for making it. It\ view qf these cpnsiderations. it is often . 
ne<^sary to have* the Courage 'to sacrifice academic^ zeal >when 
priorities have changed or objectives have already.ibeen reached. In 
j> > . such cases, research and/or evaluation are ho longej; necessafy, since 

* . they may only show that the level of achi^ivement was 93.2 percent 
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instead of 93 percent— in such instances, these projects are no longer 
relevant. 

In order to reinforce this statement, we may add that water 
fluoridation has not been initiate in many countries because of an 
alleged lack of resources for thg/ evaluation of such a program. In 
reality, further evaluation is completely unnecessary, since the results 
are easily foreseen (a 60 to 65 percent decrease in caries); further- 
more, the cost of the evaluation might be more than that of the 
program itself. 

Relationship Between the Adnninistrator and the Evaluator 

The relationship between the administrator and the evaluator 
implies a process of communication and feedback. It is essential that 
the administrator trust the work of the evaluators— how they do it 
(methodology) and why they do it. Frequently, Researchers complain 
that the results obtained are not considered by decisionmakers. This 
is sometimes the case, but there are usually good reasons for such 
behavior, and they are not the sole responsibility of the administrator. 

First of all, some researchers seem to ignore the times we live in 
and work on problems that have very low priority when compared 
with all the other problems of the country or region. These workers 
must learn that even if their studies kte very interesting and their 
methodology perfect, the results will be of no use to the country 
because they are not timely. The' researcher must live in his own 
time, within his historical moment, to undersUnd it, and to focus on 
the solution of existing problems. If he blocks himself in an "ivory 
tower," he will be unable to influence the destiny of his country. If 
his arribition is to participate in and influence the change process, he 
must abandon' his seclusion and become aware of reality. In this way, 
he can plan his research study so that the results yield alternatives 
.from which the administrator can choose the most feasible and 
suitable one, that with the greatest potential impact on the population 
to which he is ultimately i^sponsible. 

'^^'In (dealing with those projects, surveys, and evaluations whose 
objective "is the study of a specific situation in a. given ^roup, one 
must rememj^r that they should be promptly implemented. Other- 
wise, the results might not, be applicable, even though they may seem 
very interesting. For example, in a constantly changing situation, a 
survey that takes 3 or more years (1 or 2 years for analysis and 1 year 
for publication) becomes obsolete by the time it reaches the person 
who could make the necessary decisions because, by then, the group 
urt'der stuxly has changed. Decisions made with a 4"^lay of 3 or more 
years — when conditioij^s are totally different — could be unacceptable 
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to or even rejected by the <jommunity, whose attitude would be 
justified since the decisions would have no bearing on conditions at 
the time. 

Now let us look at the ^her side of the coin. When the well-known 
"Human Resources and Morbidity Survey** was made in Cdfbmbia in 
1964 (Ministeria de Salud Publica 1967), there were many Criticisms, 
One of^hem was excessive cost, I had the. opportunity to contest this 
opinion in M^racay, Venezuela, when the preliminary results and' 
analysis of the study*s methodology were presented for the first time 
to researchers and administrators in this hemisphere. Just one of the- 
findings has halted the construction of a hospital whose cost would 
have been 15 to 20 times that of the\tudy. With this finding, the 
study was more than paid for, and 'anythrag left over was considered! 
a marginal benefit or an investment profit) 

Oq the other hand, the review and ar yjysis of the life expectancy 
table in 1967 permitted the CovernHien^o sh^w the Colombian labor 
sector that calculation of the retirement aj/e had been based on 
inaccurate life-expectancy figures (Mii\ ^stej yfqe Salud Publica 1968), 
It is very difficult to determine the benefit of this discovery to the 
country*s economy; between that year and the present, the figures 
might have added up to billions of pesos. 

How can the admini&^nator or politician-administrator and the 
researcher reach agreerfient ort the objectives and content of a 
research project? As one approach to this difficult theme, I suggest 
th^t we start with the assumption thftat the politician-administrator is 
not necessarily pursuing ignoble ends. He wishes to succeed in order 
to satisfy his personal vanity, or to gain votes in upcoming elections, 
or for some similar reason. In my ^opinion, all these reasons are 
acceptable if they coincide with an intention to improve the living 
conditions \>f the 'population. Furthermore, if we have a researcher 
who is "in ftunc with the times** and is aware of the problems of his 
society, then it is not difficult to reach an agreement on which studies 
should be catried out and on the amount of time and reso]firces that 
should be allocated for such purposes. The studies should deal With 
research that has immediate application and with research that is 
"pure"^in other words, research that may not have an immediate 
application but will give research methodology a more sophisticated 
approach and will also raise issues on problems that will be of major 
concern in 10 to 20^years, 

A study on the change in environmental pollution in our cities 
could be an example. Some years ago, a proposal to research this 
problem might have been received with skepticism. Today w^ realize 
that pollution is becoming worse, necessitating corrective measures 
that could have been avoided by appropriate location of factories and 
alternative solutions to mass transport needs, for example. 
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The important issue is that the administrator realize the signifi- 
cance of a successful research or evaluation study. He often fails to 
understand th'is significance, and we hear complaints from the public 
of skepticism toward evaluations and irrelevant information. Ongoing 
communication between the administrator and the cvaluators is also 
necessary so that interim results can be utilized as appropriate. 

In an effective interrelation between an administrator, who must 
face the problem to be solved, and a researcher, who often ignores 
the practical aspects, certain facts can be identified so that the 
evaluative process can be assessed during the initial stages. In this 
way, appropriate activities are supported and problem areas can be 
readjusted or eliminated. :For example, during the initial activities of 
a study on r|ial nutrition and mental development (Mora, Ortiz, 
Florez, et al. 1976), it became apparent that 65^ercent of the 
mothers did not possess a clock. Since Bienestarina* must be cooked 
for 15 minutes, before being given to the jchild, a timepiece was 
necessary for accuracy. Oncetthe project's director was informed 
about the problem," appropriate modifications were introduced— 
evidence of the benefits of good communication between evaliiators 
and administrators. 

The evaluation process has been considered as a procedure which 
utilizes scientific methodology, but we must be warned that on many 
occasions the opinions of the community, as well as the participant- 
observation process, sometimes do not receive due credit. In many 
instances, the usefulness and relevance of the community's opinions 
have been proved in making decisions that deal with reality — such 
were the cases which arose when the Rural Promoters Program was 
established in Colombia (Ordonez Plaja 1967, p. 42), 

Conununity Participation in the Evaluation Process 

It is impossible to discuss evaluation without considering the 
importance of the community in relation to the process. Its signifi- 
cance goes beyond the fact that in most cases the community is the 
"target population." For true participation, the community , must be 
involved during the first stages of programing and design. Commu- 
nity participation will add validity to the process. Furthermore, the 
gradual learning process that will occur as a result of the interaction 
will eventually deqfipnstrate the usefulness of evaluation. Realizing 
the importance of giving correct answers when surveys are 
carried out, of keeping appointments, of participatiJJl and so on, is 



*Bienestarina is -a mixture of high-proiein-contcni cereals plus vitamins, minerals, 
milk* ;»nd so on. 
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part^of a process which sooner or later will have repercussions on the 
community members in an increasingly favorable way, inasmuch as 
their participation was requested in a serious, honest, and timely 
manner. Their participation has other marginal advantages, suqh as 
developing the researcher's awareness that trXith is not his sole 
property— thereby dispelling an image of "the almighty know-it-all" 
who can solve all kinds of problems from one point of view while 
ignoring or even rejecting the values of the participants. 

We often hear that a certain action has been carried out with the 
community's participation when, in reality, 'the community has been 
used or manipulated as the decisions are being implemented, It is 
obvious that the community feels that its human dignity has been 
affronted when a decision tlTat affects its members is taken without 
their participation. B^t it must be even more irritating for the 
community members to see publicity announcing their involvement 
in the progrg^rti ^when they have not even been initially .consulted on 
their percept'iofn of the matter^ These perceptions can' ;be right or 
wrong; in most cases they are.' .right. Wliere/f^^^^ is in 

determining what measures should be .t^ken to solve):hfe problenfi. 

The following situation illustrates-fhe n^ed for conamuriitjK partici- 
pation. In a small village, the coramunity^reqUested a chil<5pen's 
hospital to combat the high infant mortality. i^^?djagp««jsis.w 
but the proposed solution was neither appropWkle ^^npr liable. 
Discussion with the community members showed thorn thia^t ♦ Tna>;^)e' 
the installation of facilities for potable water, ,addption of a few 
sanitary control measures, inspection of food, and formafor informal 
education in child feeding could drastically decrease mortality. 
Through a participation process, the community accepted solutions 
that were technically more feasib|fe, but this acceptance was achieved 
on the basis of solid arguments and with the participation of the 
community in the decisionmaking process. It has been said, "Nobody 
knows better where the slibe hurts than the owner of the foot." This 
was the case in these communities: They knew very well what their 
problem was, even though they di^ noi know the appropriate 
solution. 

Evaluation and its Financing 

The funding of research or any other method of evaluation where 
external financing is involved creates situations that are worth 
analyzing. Quite often, persons with wide experience in other-' 
socioeconomic ^nd cultural environments arrive in a country with 
preconceived ideas as to what should be investigated and how to 
evaluate the investigation. This point is very important, ^ince the 

^1 
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evaluation, even though it may seem necessary from a scientific point 
of view, can overlook the human resources, habits, and traditions of 
each country. Tmis an evaluation which is almost perfect in method- 
ology may not meet the real needs of the country* Galbraith (1971) 
has suted: "We are inclined to believe that we are becoming much 
more scientific about the whole business althcjugh in an established 
tradition oPthe discipline there i& some tendency to identify scientific 
precision with mechanical elegance rather than reality/* 

When we complain about research and methodological systems that 
are proposed or imposed on us by donor agencies that are aware of 
our inability to fully implement these systems, we are facing a 
problem which is basically the same as the one mentioned before, 
even if it is on a different level. 

Proportion of Resources Dedicated to Evaluation 

How much of a program's budget should be allocated to evaluation 
purposes is frequently discussed. The answer is not a simple, one; a 
definite amount cannpt be arbitrarily assigned, because there is 
interplay between a large number of variables. For example, when we 
are dealing with a program about which there are serio^us* d.ouSt$', we 
might very well assign a surii that might secrn quitrf high (ot\ 
comprehensive evaluation of the program's usefulnesi?, convenience, 
and acceptance by the community^ In such a case, tl^e investment li 
worthwhile, even though it may seerini excessive. ^HpweVe^, as men- 
tioned in an earlier exaniple, it V be absufd t6 invest any 
amount, even l^^percem.. of the %dial buclget erf a wdter fluoridation 
program; to: detei^tnirie the usefulness of fluoriclatipn. The sanpe ^ 
could be *said about, studying the impoftance^of, breast feedingp^iftlv 
successful results need no further scientific confihnyalion. 

In programs benefiting children, where the Yc^qlts are obvious arid ! 
where action is the required element. I^do np^ b^lieVe that a defidite 
percentage figure ought to be assigned to program evaluation/Each 
program should be dealt with according to its circumstances. In ihis 
case common sense is of more valuer thaii any scientific-formula Iv 
could propose. In the wordi of Einstein^^ "Imagination is ihbye 
important than knowledge.** * ^ ' j . " ■ 

In summary, what the administrator, wants is relevant; reliabl^i ^and - 
understandable inform'atiori that will allow for sound decisionmaking' 
The scientist should consider this n(^ed, especially when presenting:^ 
results that ipay be very clear to hijn, but not to 'the adminiStra|tor. 
The researcher-evaluator must rert)e*mber .that his conclusfons: ^^ 
only be given credit or taken into account if they are in accord^rice 
with the real needs of the people, and he must rememb(^r that \the . 
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decbionmaker is genuinely willing to solve the probicdn and usually 
has the necessary resources to do so. 

I'inal Comments. 

Tp close the discussion on this subject which in its very essence 
needs a balance among science, humanity, politics, and facts and cases 
that improve the quality of life, I dp not find any more appropriate 
words than those of Eduardo Cote Lamus, one of my country's 
greatest poets: 

4 

That world became extinct 

thus hadito bum up that way 

because the things dear to man were destroyed: 

his will, his faith, his attempt 

to become one with his fantasy. , * / 

And he was left with only the ability to reason.. 

We cannot pretend that science is the only form of .k\«p>ylpdKe or 
that its conclusions are the sole tuith; they are ift^Klrtiui^, tsiiUt c(i^jy as . 
. a part of something bigger and more complex — -tKe quality oIF life of 
individuals and communities, their expectations, their visions of the 
world, and their values. These factors are such that the administrator 
cannot solve a problem without the help of the iscientist and vice 
versa; nor can^both solve a problem without the participation of the 
beneficiary (or the legal guardian, when we refer to children). . 

In today's world,, the; politician-administrator must'*Wkjr^ 
that he itee^s the heip^ofscrentists arid techjlical a4,Visiprs if his; goal is 
a better ^hance pf succels. But the scientist must-aisd'^^iinderst^ that 
the extent of his participation in decisionmaking (a new-phertomenoti 
in the world's history) (Drucker 1968) implies a great responVbility, 
and under no circumstance should the scientist consider himself to be 
-above the f)olitician, since the latter is ultimately responsible to the 
jpieoplc. This advice is valid in any political system, and although this 
i^^p^sibility can and should be shared, it ultimately rests with the 
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EVALUATION OF CHILD HEALTH CARE AS SEEN BY 

EVALUATORS 

Jaime Arias 

Introduction ^ - 

There is a logical order to the three previous presentations. To 
continue along the same lines. I will present some ideas^on evaluaUon 
of medical care from the viewpoint of the evaluator. Dr. Wray and 
Dr. Rosselot have described quantitatively the problems affecting 
child health in the Americas and have summarized the existing 
programs created to tae(c.1e those problems. They have identified the* 
problem for us, allowing us to proceed to the conceptual, political, 
administrative, and technical considerations in order to evaluate both 
pro<;ess and outcome. Although identification and quandfication of 
problems and theii^ related health, activities are components of the 
evaluation process, these aspects cahnOt be considered evaluation in 
themselves unless other aspects are included, particularly those that 
permit comparison and judgment of data and information. 

Dr. Horwitz has pointed out the importance of child health 
programs compared with programs devoted to other risk groups. He 
has further directed ou^ attention to efforts to measure and evaluate 
programs in the field of health care in each country of the Americas. 

One of the basic purposes of evaluation is to determine priorides 
between programs; the priorities may then be^examined in relation to 
the overall economic and social development of the country or 
community under study. These considerations are more important 
when one is dealing with a counti;y or community where scarce 
resources must be distributed optimally, avoiding waste and maximiz? 
ing the output of each resourc^. 

Before priorities among different health programs can be set, 
country must det^mine its basic policies, such as the relative 
importance of its investments in education, nutrition, and medical 
care: Then, within the health sector, the i>olicymakers should deter- 
mine if, for example, preventive medicine programs are justified in 
light pf the coUntry*s existiQg needs and resources. What^ is the 

■ N '57; 
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proportion , of primary care required vis-a-vis secondary car^. and ■ 
how can we use evaluation tools to find out? . ^ 

The aforementioned considerations bring forth a multitude of 
questions: What aspects of a program are 'to be assessed, and how? 
Which activities should be evaluated., and what measures shoyld be 
used? How can one assess efficiency, cost-^benefit and cost-effective- 
ness, p^j^gram efforts and inputs, quality of procedures, and. of 
course, short- and long-term impacts of efforts? 

Dr. Ordo nez Plaja has discussed evaluation in terms of appropriate- 
ness. More importantly, he has shown th.^t there ihust be an 
appropriateness in the use of results. He has also spoken of the • 
factors that can decrease effectiveness in the relationship between 
administrator arid evaluator. He has emphasized the time factor and 
the importance of clarifying investigative and evaluative goals. 

Community participation in the evaluation process also been ' 
meritioned; the community should participate, not only as- the subject 
^ of the evaluation, but also as a key /actor in the evaluation process 
itsejf. front the programing and design, of the eyalOation to the 
irnplementation of change. • ' . . ' ^ A 

Dh Ordonez Plaja explained thjjt external funding sources rrtay 
influence the evaluation {)rocess ^ccbrdin^ toaheir ,own interests 
rather than those of the program or agepcy under study.- Heqcc a 
certain amount or proportion of. funds should be set aside^for 
^valuation in each particular agency or program. His viewpoint is 
that of the administrator at the jxJitical level for whom evaluation is 
just another activity belonging to a broad planning and administrative 
process. . * * 

'As a follow-up to these thoughts^ I shall disquss three topics:;. 

' ' . » 

— The definition of the evalu^^tion process within the health care 

system; . 
— The ;*evaluator.** his or .her role, field/ way pf working, and 

relationships with politicians and administrator?; ' > ? 

—Technology transfer in evaluation betweeri tlte countries of the 

Americas and the future 6f evaluation of health care programs in 

ihereg^n. 

Evaluation as Part of Medical and Health Care 

In the last deciide evaluation has become so important it is in 
danger of becoming an end in itself, an independent system, when it 
, should be an integral part of the planning of medical and health 
care. * ^ 
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Figure 1. Flow chart of the medical care process V 

Using a so-called Health-Disease-Intervention (HDI) model, one 
can apply the different aspects of the evaluation ptoceSs in order to 
clarify the relationship between health care aind evaluation and to 
illustrate the relevance of the latter in that general health care 
context. 

. Figure 1 is a flow chart of the overall medical care process, 
including a populatiqn at certain risk to disease and the requirements 
of that population for medical services; a network of individuals and 
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Figure 2. Normal rWionships between health care: and evaluation processes 

institutions organiicd to render. sefwces tb^4ugh a series of specific 
procedures for ea^h patient ani^partkrubr disease; a financial 
infrastructure that Wludes carts, prices/ fees, and ^payment of 
services.. The infrasWuc^ure/is lloca^^^^ at both the level -of' the 
population and the leV^lo^jthe iiistitttUons and individual providers 
Finally^ the interaction ofdemandj arid supply (or capadty of services) 
has an outcome affecting individuaj's» the total population/institutions 
of health care» individual providers^ the specific health system, and 
the global social system. d 

Figure 2 shows the different types of evaluation which have a clear ' 
relationship to the overall process of medical care in such a way that 
each of the components of that process can be assessed according^to 
a complete and systematic scheme. ' " 

We can see that some types of evaluation require only jhe 
consideration of one component of the medical care process. Others 



s KvaluAtioit .As SetMi by t valuators 61 

imply consideration of both demand tor and iuppl^-of services, 
- Considcratioii of outcome, as in the case of impact analysis or cost- 
■x benefit studies (including the comparison of outcome' with the cost 
an(J effect of the projjrams), is also important. * 

There never will be one ideal evaluation ni^hod, but wc caa hope 
to deveh3p adequately conceptualized and executed methods that will 
vary according to the appropriifteness of the process to the program 
to be evaluated,. Only when*lhe various evaluation aspects are carried , 
out can we have adequate and sufficient knowledge of a program. 

In real life, complete and absolute knowledge is neither possible 
'nor necessary. The accumulated knowledge about most components* 
of the HDI process permits the application of programs and technical 
Xprocedure?. In many instances, the causes and mechanisms of diseases 
^re well established, and duplication of basic /research is not needed — » 
.'particularly in the less-cleveloped countrfes. ^ 

'Basic epidemiologic studies do not require duplicattori in each 
country or in each particular situation, whereas applied epidemiologic ^ » 
studies should be carried out for each different situation, even in 
less-developed countries, since the complexity »and costs of »such 
studies are hot beyond the ability of .siich countries. One example of 
such studies would be *'The distribution and prevalence of kwashior- 
kor in a given rural province of a country^" 

When one analyses the secondary, stages of the HDI process, it is 
clear that specific evaluatio.n studies are needed— not only for each 
country, 6ut als^;for each program, or at least for each particular 
type of program— since a number of social, cultural, economic, and 
psychologic variables influence the effectiveness andj impact of the 
program or procedures. This type of evaluation is essential for each 
country, program, andcinstitution. An example of this type of study 
would be "The ^effectiveness of pt'9gram X (or procedure YY in 
decreasing the prevalence »of kwashiorl^or among a given population 
of children." » 

Evaluation of clinical, surgical, laboratory, or other specific proce- 
dures, is done at the tertiary stage of the proce.ss of organization of 
.serjk'ices. This sequence , doe? not mean that effort evaluation or 
effectivene.ss analysis, for example, applied in previous stages is more ' 
important. 

^Figure 2^^jord table 1 give Us a better understanding of the prirtiary, 
secOndary;^;and tertiary stages of the HDI process. They help us to 
understand the r^fationships among basic medical, research, epidemi- * 
ologic studies, ,|irbgram evaluation, apd clinical* procedures evalua- 
tiqft..' ' , V • ... 

s^raditionaJ" definitions of evaluation do not include basic research 
causes^iS^nd mechanisms of cTiseases, therapeutics, oi* epidemiology. ^ 
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However, when these types of research are ^considered ip relation to 
the complete HDI process, one can understand their links to 
traditional evaluation studies. 

In terms of evaluation methodologies, we will Kave the opportunity 
to hear and analyze^nine cases, selected in such a way that we will 
have a sample of most of the best known methodologies^^or various 
program needs and circumstances. It is important to understand the 
adaptation requirements of evaluation methodologies t6"*speciric 
program needs. We can identify at least three different levels: (1) 
experimental or quasi-experimental designs of varying complexity; 
(2) nonexperimental designs for program evaluation; or (3) very 
simple designs for self-control of performance, suitable for any 
practitioner. 

Table 2 is a diagram of the three possible stages of an experimental 
program or procedure. The first level requires experimental x)r quasi- 
experimental dc^signs, with control groups, and so on. This stage has 
been called ^'evaluation research." Once the programs or procedures 
have been evaluated, they can become "demonstration programs or 
procedures." These in turn should be evaluated by means of a 
technical and sometimes* complex design that is, iri general, of the 
nonexperimental type. Finally, when demonstration programs or 
procedures have proved uheir appropriateness add feasibility, it is 
possible to multiply them^ as "extended progr^s pr procedures." 
These require program evaluation of simple dei^gn and easy applica- 
tion. This type of evaluation can be called 'Control evaluation" or 
"monitoring.** 
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The Evaluate r-r-His Role, Work Setting, and Working Relationships ^ 

.-V ;P ^. 

XJijce th4^valuation proclSis is defined, it is ri^Eesj^ifcy to ^cide 
who will carry it out. First, it should be ascertained whether there al^^ 
ayailable professionals wljfir. are devoted •exclusively to evaluation aijSl^ 

have received formal training ia^he field, or *vhether there are 
individuats ^who have learned how to perform the functior>s of an 
evaluator. Second, the evaluatpr's role, the setting in which "fte 6r she 
'carries out that role, and the relationships between the evaluator and 
cidministrdtors or decisionmakers rniist Be considereci, - 

A banel of three esteemed medical evaluators has reviewed^^the * 
more important^tvaluation studies published in Bogota, the ca^itarof 
Colombia, during the last 2 years^and it lias selected 30 individuals ' 
who participated in the studies and who. can be considered as 
evaluators, or at least temporary or partial evaluatiDrs, A simpde > 
survey'was distributed among rfit grorup 6f^30, rg^uesting informa- 
tion about their professional backgrjound^, a^tivitiesf and ^the^etting ^ • 
arS3 conditions of their work. None'^cohsidered sjpyma tic evaluation , 
to bfe his/her only-^^professional^llictivity, and o^ly flif (16.6 peraslat) 
said evaluation was the nwst important of their professional a^ivities. 
Only three evalua^jUrs at'the tirne of the survey vVere \j^ot*king on the 
discovery or testing of new indicators. ^ « n 

Twenty individuals (€^6. 6 percent) had rec^ved adequaft postgrad-^ ^ 
Xiate training in» evaluatio^j^lmost always as part of tr&injp^g in public 
health, sociology, or statistics; the^est had been trained^through their 
professional practice.^Of' those with formal training, 1 1 <33.0 peVcent) - 
had studied in tfte United^tates or England for more than 2 years. J[ 

In Bogota the majdri^^oP evaluaU)^ s of health Jteograms are^ 
medicai doctors (23); the rest are soci(9feists (3). statisticians ^),^nd ^ 
economists (2). Within the grt>up of 83 medical doctor^ 15 had 
different Specialties in, public health, social medicine, cit community rj. 
healthy and the. rest were clinicians. ^ J- 

Taking into account the amount of professional time devoted to A 
evaluation* during the last' 2 years, the survey revealed that 7 of the ^ 
respondents* had spent more than 4 percent ^j^^them time on 
' evaluation; 4 sp>ent between 20 perceutwand ^0 p^-Vceni; and the other 
19 spent less than 20 percent of |heir time on evaluation. Twenty- 
three were carrying 6ut s'orrie evaluation study 4I the time of the 
surveyT and only 13 had newly designed and funded evaluation 
project^ for ^the next 2 y^ears. Twelvi^ worked for govern me ntJjl 
institutions (40 percent); the remainder, for universities or research^ 
organizations. - n 

It is clear, at least in Latin America, that there is n6^ a Special 
group of professionals devoted entirely to evaluation of medical care, 

' ' - . ^ 
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'and on* can predict that this situation will not change ir> the near 
future. Since the "felt need" for evaluation is increasing and most 
programs require it, it is likely that the number of individuals^ 
developing new methodic and indicators, a^ well as those doing 
evaluation, will increase— but qot to the extern to which the profes- 
sional qip be considered purely as an evaluator. 

According to presenf trendy we can predict the need to train at 
least three types of evaluators for the future: 

— Those deypted to the experimentation and development of metho- 
dologies and indicators that will be applicable to extended pro- 
grams, projects, and individual practice. These professionals will 
work exclusively in %valuatidh, and their SUpply will grow more 
rapidly in industrial countries: . 

—Those individual devoted' to the development and evaluation of 
exF)erimental, pilot, and demonstration programs or to the evalua- 
tion componept of extiended or action programs. These individuals 
should. have formal trafning in evaluation, i 

—Finally, those pfc^fessionals vrfio will perform some sort of self- 
' evaluation, of their activities; therefore, it will be necessary to 
include the teaching of basic evaluation techniques in the curricula 
of most health professions. (Toiday, the health professional does 

. not have information about evaluation, yet most medical and other 
health-related schools are not including information on evaluation 

^n their curricula.) - , , 

Now that I |iave briefly .described the characteristics and profes- 
sional background? of the present Latin American evaluators, I will 
attempt, to identify th£:J». roles and the conditions- of these roles. 
Evaluation is refttively new^in Latlfi America, an<the evali^to^'s role 

not well define^. It varieS accortJmg to the type of evalwation,^the 
program^ to be evaluated, the purposes of evalaation, and th^ institu- 
tion w^ere the evjtluation is takittg "pla^— that is,^^ccording to the 
^valuator's working conditions. 

\ When i^aluation tajces place in afi-acadcnfie settirigj, researchers 
ha>^ more freedom and con^bl over theirtwqf^k. The reverse occurs 
when professiorthls are doing t^j^eir work for an '^gency devoted to 
health progtams. The rJle also varies accprding lb whether an 

«^evaluator is working on a single project or progfam or for an 
evaluation unit in charj^ of the coondinkion of many projects or 
prpgram^t. * ^ 

in^the case of pn)gram Evaluation, the role includes the evaluatoi^ 
participatu|n^ in planning and program design, as well as^in <he 
implemenWtion o| changes through the application of evaluation 
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Faults. In the case of control evaluation or monitoring, the evalua- 
tor's role should focus mainly on the teaching and advisory aspec^s^ib 
order to help those individuals who are doing self-e.valuation. 
In general one can summarize the evaluator's functions as follows: 

— ^To promote evaluation techniques; N-^ 

— ^To participate in the design of experiniental programs or promts; 

— To design an evaluation methodology to be applied to the program, 
ar>tb select one which is appropriate; 

— Tp carry out direct collection and processing of data and informa- 
tion needed, or to advise on data and collection methods; 

— ^To analyze or to help in the analysis of collected information; 

— To descri^)c, explain, and communicate the evaluaticJn results; 

— In some cases, to motivate and persuade the administrators con- 
cerning the importance of the use of results in the change process. 

—To collaborate in change implementation, basing it on evaluation 
results; 

— To educate those individuals engaged directly or Indirectly in the 
^valuation or planning processes. 

When the evafuator's functions are taken into account, the impor- 
tance of the relationships betweenr the evaluator and all those 
individuals who plan, manage, and make decisions on projects or 
programs becomes apparent. The success here depends not only* on 
the adequate execution of the evaluation, but also on the capacity of 
the evaluator to "sell*' the results and engage all the key persons in 
this complex process. 

Thus it is important to consider the management of int^erpersonal 
relationships. The evaluator should detect and analyze carefully the 
constraints and determine which individuals are more likely to ' 
obstruct sound relationships. Taking steps to avoid or to detach and ^ 
solve these problems is part of the strategy that must be developed at 
the beginning of any evaluation effort^ 

I would like to stress the important role of evaluators as educators. 
This role enables evaluators to multiply their efforts and permits the 
^ijtutionalization and interniilization of the evaluation concepts. The 
.St tWQ papers of this conference address the importance of evama- 
tiori as a tool for education. 



Technology Transfer of^valuatipn Between the Americas 

It is clear that evaluation techniques are developing at different 
paces and in different directions in -industrialized countries and in 
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preindustrialized ones. We mentioned before that basic and epidemi- 
ologic research should be the primary focus of developed countries, 
while evaluation of experimental, demonstration, and applied pro- 
grams, as well as the evaluation of specific procedures and applied 
•^epidemiologic studies, should be of interest to both types of countries, 

Vye should encourage technology transfer and communication in 
those studies common to either type of country. Transfer should take 
place both 'at the level of new methods and indicators and at the level 
of results of thos^e analyzed programs, : 

Transfer and communication should occur not only between 
countries at different stages of development and with different 
♦cultures but also, more importantly, between countries of similar 
development and similar problems, and between institutions or 
programs' within a single country. It may seem obvious, but it is 
important to reiterate that in countries with technological and 
financial limitations, it is very important to avoid duplication of 
studies. The ready exchange of ideas on methodologies and results 
will eliminate this problem. • . 

In order to establish effective transfer, it is necessary ta create new 
communication mechanisms: conferences such as this, publication of 
research, studies of different programs, and, training of evaluators 
with participation of different countries. 

It. is desirable, through an exchange of ideas, to'devise simple 
methodologies and indicators that assure validity and reliability and 
a,re simple to execute. Such designs can be applied more universally. 

It should be noted that this paper does not make specific re^ence 
to evaluation of child health programs because, from the evaluat<^*s 
point of view, the principles and techniques of evaluation are similarly 
applicable to any population-at-risk group. 
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— Purpose of evaluation and originators of the evaluation; 
— Type of evaluation: effort, efficiency, effectiveness, impact, cost- 
benefit, quality assiessmenti 
— -Methodology and indicators used in each study; 
—Use and implementation of results. 

The workshops analyzed each study primarily in relation to four of 
the above-mentioned characteristics: the type of evaluation, particu- 
larly in* relation to the objectives of the program; the problems 
encountered with the methodology; the difficulties ii\implenfientation 
of results; and the applicability of the method to other programs and 
cultures. 

There is no particular technical reason for the sequence of the case 
presentations. The chosen order intermingles Hispanic and English- 
speaking authors in an attempt to enhance communication among 
professionals with different janguagies and different cultures.. 

Based on the above, the case studies can be classified as follows: 

— Country Ju origin. There are four studies from Latin America and 
the Caribbean (Agudelo, Heredia, Karnsey, Yarbrough); four from 
the United States and Canada (Belville, Ressner, Pless, Williamson); 
and ©ne from the United Kingdom (Martini). The paper from the 
United Kingdom was presented by a Latin American investigator, 
while ohe of the Latin American studies was presented by a North 

, American investigator (Yarbroug:h). 

— Size of the geographic area. There is substantial variation in the 
programs under study. Two na^ipnal health j^are programs are 
evaluated (Heredia, Ramsey). THe remainder focus on counties or 
provinces (Agudelo, Pless); limited communities .(Kessner, Martini, 
Pless); different forms of practii(lBj(Kes|^ier); health centers 
(Belville); and, in one case, a variety of^ttings^(AVilliamson). 

— Jype of population-at-risk. Aln^ost all the sti^dies are related to child 

» populations. In those not directly related to children, the method- 
ology is applicable to child care programs. In one instance, the 
evaluation includes mothers as we|l as children (Heredia). 

— Level 'or area of health care under study. Some papers stress, prim 
care (Agudelo, Belville, Kessner, Martini), and others focus 
secondary care or on all three levels of health care (Heredia). Twx) 
cases refer specifically . to nutrition (Ramsey, Yarbrough), and 
others use specific diseases as indicators (Kessner, Martini, William- 
son). 

—Type of program. In some case studies, the evaluation has been 
applied to "experimental programs," and hence ,their design is of 
an experimental type (Kessner, Martini, Ple5s, Williamson, Yar- 
^ brough); other case studies are related to demonstration programs 
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(Agudelo, B.elville, Ramsey) or to the so-called extended or current 
programs (Heredia, Ramsey). 

—Purpose of the evaltuition. A farge proportion of the papers have an 
academic purpose, and their initiation must be credijled^'to acade- 
micians. In' many of these academically oriented studies, the 
application of research findings to program development is a clear 
aim. In some instances, the study goes beyond assessment of a 
particular program or project and seeks the discovery and devel- 
opment of new rhethodologies (Heredia, Kessner, Martini, William- 
son). Some of the papers have a clear administrative or policy- 
making purpose and attempt short- or long-term program changes 
(Belville, Heredia, Ramsey), whereas others rely more on educa- 
tional processes to achieve change (Kessner, Williamson). 

—Type of evaliuition. Some cases address issues of quality assessment, 
including elements of effectivene.s.s (Kessner, Martini, Williamson). 
Two case studies address program effort or inputs (Agudelo, 
Heredia), and other papers focus on studies of program eff\cien<J|r 
and effectiveness (Agudelo, Belville, Pless, Ramsey, Yarbrough). 
One case includes cost-beaefit and long-term impact analysis 
(Heredia). . y 

— Methodology, A wide range of methodologies and indicatic/rs is 
utilized in the nine case studies under analysis. There is a 
mathematical simulation model (Heredia), which, y3es known indi- 

. cators of outcome, effectiveness, and efficiency. There are experi- 
mental designs with well-selected cbntrols^(Kessnef, Martini, Pless, 
Williamson, Yarbrough) that use specific health conditions as 
indicators with a different methodological approach. Martini uses 
sociomedical indicators with d*a provid^j^ by the patients. Kessner 
uses a prospective approach for analysis x)f outcome which he • 
combines with education. Williamson analyzes the diagnosis and 
treatment proce.sses and their outcome for speci^Qc health^ condi- 
tions and relies on educational strategies to achieve early change. 
Pless presents a more global approach to the evaluation of a 
community health program using systemic surveys as a way to 
assess health status and services rendered, and Yarbrough presents 
♦a quasP-experimental design to measure the effectiveness of 
nutritional program. Belville uses a method called "concurrent 
evaluation," which attempts to produce ongoing change in the * 
health care program through active participation in the study of 
those engaged in the operation of the programs. Agudelo and 
Ramsey combirte the measurement of effort and effectiveness in 
demonstration programs using nbnexperimental designs and rela- 
tively simple methodologies v^th a combination of indicators. 
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— ImpUmerttation of results. In sOme of the cases, the use o\ results is 
direct and immediate (Belville, Ranisey, Williamson); in otherj, the 
use qf results is direct but requires some: delay i" their implemen-*- 
tation (Agudelo, Ressner, Martini, Ple§^)t In the remaining cases, 
the use of results is indirect and requires a longer period of time 
jbeforc implementation (Heredia, YarbroUgh). In some instances, 
the application of results is made directly Iby an individual practi- 
tioner or by a group practice (Ressner, Martini, Pless), v^ereas iri 
other instances, the application is made by the Government or 
administrators (Agudelo, Belville, Heredia, Pless, Ramsey). Finally, 
in several papers the method in itself and the possibility of its 
application are the main fociis of the study (Heredia, Ressner, 
Martini, Williamson). 



SOaOMEDICAL INDICATORS FOR CHILD HEALTH > 

Carlos J, M . Martini and Ian McDowell 

Introduction 

Many indicators have bcpn used to estimate the need for child 
health services. These fnclude birth rates, the incidence of low birth 
weight, the proportion of th<&' population under a^e J 5, stillbirth rates 
and, most important of all, infant and perinatal mortality rates 
(Report of the Committee Oi> Child Health Services 1976, Lowe 
1976). In addition to these, a variety of other indicators related to the 
mother,, such as the number of children she has, short pregnancy 
intervals, small physical stature, socioeconomic status, and certain 
behavioral variables, such .as whether or not she smokes, are being 
used in Britain as bases for high risk strategies in planning care for 
children. . ' 

Although these data are used frequendy, some caution mUst be 
expressed over their use as indicators of need and for evaluating the 
outcomes of child care (Department of Health and Social Security 
1976). While indices such as the sickness and death rates of children 
at birth or in the perinatal period may' provide useful data fon the. 
overall planning and development of social and medioi^ services, they 
arA not specific enough to identify the efTectiveness pf health care 
alone. Infant mortality rate's in general are Rriown to reflect a number 
of social and environmental influences. They should therefore be 
considered as overall indicatcrs^f community poverty or deprivation 
(and therefore, perhaps, also as ee^mates of global needs for services), 
rather »,than as specific indicators of the outcbmes-of healthu services. 
We have examined the relativje contribution of medical care provision 
and socialr factors in explaining the interregional variation in a 
number of indicators of child mortality and morbidity in England, 
and Wales (Martini et al. 19J7). Figure 1 gives a summary of the 
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Figure 1. The relative contribution of medical care and sociodeniographic 
variables to variation in two indicators of child mortality rates, England 
and Wales, 1970-72 , ^ ^ 

typical findings, which show that the proportion of variance in infant 
mortality rates explained by sociodemographic tharacteristics is more 
than twice that explained by medical care.' ; ' 

For perinatal mortality, the relative contribution of the two com- 
ponents was' virtually equal, but the data show that this indicator is 
still relatively insensitive tq the provision of medical care. Similar . . 
'results were also obtained for crude and specific mortality rates. 
However, one indicator — the overall case fatality rate— appeared to 
reflect the contribution of medical care more adequately. In this 
instance, the rhedical care contribution to variance explained was 0.56 
(the total variance explained being 0.75). » 

Let us assume that these findings express, albeit crudely, a real ^ 
phenomehonT— that the impact of medical care on manry accepted 
indices of child health is siibordinate to that of the socioeconomic 
and environmental circumstances of the population. The interi>reta- 
tion of this finding will vary according to our expectations (pr the 
scope of medical care. If, for example, we argue that child health 
care should strive to improve the total environment of children, theiv 
these results suggest that such care is not succeeding in its goal, for 
its influence is still outweighed by that of sojcial circumstances. J 
Alternatively/if we; take a more restrictive view of the role of medical 
cart, then these inclite«:rAre not adequately specific for its evaluation. 
To select more suiiat>ljf i^(d^ for planning and eyaliiating . care, it is 
necessary to dej^rit^^^ detail the various comi>onents of the 

planning prcK:esk,. ^i^^^^ ' 

J" . A •■«' " ... 



* The variables 'considered were age of the population, social class- distribution, level 
of urbanization, levels of Jhcome and education. To describe medical cai%, the 
following variables were included:, health and social services expenditure, :hospital 
resources and performance, and various indices of community care. 
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Indicators for Planning ind High Risk Strategies 

; The first stage in planning health care requires an overall descrip- 
tion of the actual Tiealth' status of the population. The previous 
findings, in our opinion, do fiot invalidate the use of traditional 
.^jjKiicqtors for this task, sirkce such an assessment is not concerned 
Nj^h specific heakJh care activities, but with reflecting medical, s'ocial, 
atid^environmental coaditions! For this jiurpose* indices of child 
morulity rates remain the most readily available and' general proxy 
measures of child health problems in a community. ' 
.•The second stage in planning care is to specify the type of 
intervention most suitable for. the problems outlined by the overall 
description of |)ealth status. With the growing emphasis on preven- 
tion, an application for .the »indice», of mortality rates again arises, 
since these do reflect social conditions. Many of these social correlates 
of mortality rat^^ - may therefore be measured in the population. 
Groups at high risk may also be identified (World Health Organiia- 
tion 1976). In this sense, the mortality rates provide a more useful 
basis for planning preventive interventions (high risk strategies) than , 
for planning specifically m^rfjca/ activities. 

The third stag^ of planning care covers evaJuation of the effective- • 
ness of the programs implemented. In evaluating c'hild health 
programs, we must address only those elements of the wliole health/ 
sickness process which may reasonably be affected by care. The 
tesults shown above indicate that mortality and morbidity rates reflecf" 
sociar realities that are normally considered beyond the scope of 
health services. It is only, perhaps, at the level of community child 
health care .that such indicators may provide relevant evaluative 
information; they are still relatively insensitivp to the detailed inter- 
^venttons pf the health care team. This phenomenon is particularly 
noticeable V-here much of the work is not concerned with {Jotentially 
fatal Gpnditlpns. I# A ' ' 

1:p; overcome these proSfems of evaluative methods, alternative 
indices of child health levels must be considered. For, example, if it 
were^possiWe to dej'elop a survey instrument to distinguish the 
cpntjibjitions of'social circumstances and of.medicallcare to a child's 
healtK it would b« possible to plan and subsequendy e'valuate health 
^re inteTventionj, tailored more specifically to existing problems. " 

on measurements' of a child's functional 
abilities, in which *e parents are asktfd about changes in their child's 
and m t^jeir own.b#havibr ias a result of the child's -ill health; could 
provide a .^fmpfe.and yA specific means for assessing the /ess serious 
types of condijiion seen ifi primary health care {McDowell and Martini 
1976). Measurements gf this type- wpulfi not be- intended to replace 
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the traditiona]>calth indicators, but rather to complement ihtsn^ first 
in the . area of evaluation and .then'ako in the assessment oPthe netd- * 
for carcV ' V , ■ ' ■ " . 

T}xi Development of a Spdomedical Child Health Index 

'.•.■'*• * * • . < ' ' ■ • ■' 

.,We ^re at present developing and testing ^ measurement technique 
in which parents (preferai>ly the ritbther) are asked 'tp comment qh^ 
;^Uerations in the child's norrnar activities and also to describe chahges 
in their own normal Fifestyfe prod;aced by. the chilcl's illHess.. (Older 
children can of courte comment d such changes.) These 

perceived changes qr problerns qah^^ used in beforer and after-<:are 
evaluation ip indicate Xhe short-term^ effects of inedical intervention 
in reducing incapacity, as weH'as in restoring th6 patientls; ftormal * 
functioning— social, emotional, and* physical. ; 

The child health \ndfik is applied through interviews with 'which- 
ever parent has brought the child to care, using a standardized * 
interviicw scl|C<lule. In. the past, the parent has been exteifsively used 
'as an. inrformaht; a number, of- behavioral inventories,"^^^|^Jr-.a3 the 
Peterson checklist (1961), 'the Achenbach index (li96^), and Rutter> « 
. screening instrument (196^),* have exemphfied thi^^-approach. The 
interview .schedule is based on a .collec;ti0n oS statements .obtained 
from 536 interviews with parents whose children were attending the 
oi4|>atient cli^ics^ the* Nottingham Children's Hospital. As "with the , 
' meihod tfsed In Bergner's Sicknfess Impact Profile (Bergner et aL 
1974), all the statemjents describe changes or restfictfons in behavior ' 
due to Ihe child's illness, relating either to the child or the parent. - 
This procedure permits the development of tw© types of profile: one 
describing^ the child's prol|lc;ms and another describing the parent's 
altered activity levels. The rationale for inquiring into the| parent's 
own behavior is . based on the simple fact that the child's illness will 
' temporarily affect his parents' ^quality of life, with possibly-d^(q^_i mental 
, consequences for the child, himself. Obviously a parent's life is 
affected by children, whether they are healthy or sick, and this impact 
is illustrated in^figure 2. The da\a shown in figurftT^L tome from a . 
preliminaty^t^Mt of the child health, index on two comrasting^ groups 
^of women: ffl^ young children (under 12 months), and 

2 women irpfi least the 28th week of their first pjregnancy. With a 
k child, the parent's normal activities will be further modified; so 
that establishing^ the patterns of activities of* mothers with healthy 
children of a given age will make i.t possible to sCUdy the. additional 
effect otthexrhiiifi's jllness.JAliernatively^^the questions can be cleafl^ 
worthed so ahat the riespdndent is asked to -feport-Gi^n changes--over - 
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StatoMtntf 

I Ultfi act l^rNiMy Immt* pMplf 
I mrry itout my itali of htaim 
I 9t4 upM( moff Milly 
I am cutllrii dMM^jnyiyilcjl KiMtIn 
I 0B not MrrtiKTwoiri ihM i itarl \ 
I Kl trrlUMy tpiwrtfi n 
t npms Norrlti |bBu< 
Mtndlocosli iln^ 
1 Nl Itit man otuai 



Gpj^i^pirw^n of mothers with young children (n » 15) and dregnar 
women (n »». 12) H / 




» 30 40 V) 60 90 



and beyond the ifermal work of charing for^e child— brought oh 



exclusively by the child's illness 
Results of Statement Collection St^e 



^4 



As the basis for constructing the child health index, 1,070 state- 
ments were collected. These were groupcc^ by four judges into ten 
categories. General titles indicatii^ the content of these categoric*, 
are shown ip table 1. In table 2, the types of statements describing 
changes iii the parents* behavjor are shown, and table 3 giv^s 
examples of some of the statements collected. 

The agc and sex of each cbUd and the tyfjjpTof clinic being Mt^nded 

were recorded. The parent's age, occupational status,\ind the number 

. \ ' 

Table 1. Tjtles of Categories of Staicmente Coriected as a Basis for Child 
Health Index:. Sections for Chiy;jlren V 



A. 
B. 
C. 
IX 
E. 
• F. 
G, 
H. 

J 



Sleepiji^ problems 
Eatiiig* probjcms 

Cha^ges in play activities and hobbies 
Sociability and £unily relationships * 
/Increased depeiraency 

irritable^ jnoody^ and un£Ooperative behavior 
EnDotionat.jind upset responses ' 
Indication^ of dep^ression and boredori} 
Languid and lisdess behavior 
"Symptoms' of ill health 
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Table 2. TiUet of C^tegoriet of Sutements Colliect'eci as a Basis for Child 
Health Index: Sectklh for Parenu 



A. 
B. 
C 
D. 
% 
F. 
G. 
H. 
1. 

J 

K. 

U 

¥ 



Sleeping problcmio 
Coping with daily routiiie ^ 
Reurictioni on going out ^ 
Changes in todal life and leisure activitiet 
Working patterhi 
Family relationihips 
'MMiHerpd caringili^M^vior and patience 
I^reased concern 
Fatigue and tiredrtHs n 
}t Increased mental tension 
^ Ksclings of depression 
irritability . ^ 

SymptoMMM health 



TabkfS;- Child Heilth Index: Some Examples of Sutemen^ Collected 



. ' liferents 
Caring Behayioi^andrPatience^ 

I am more4iMkrife?rr^fowards the 
child — afiow bad beluivio« lyithout 
tcoiding. 

When he is asleep. I check that he is 
all right more than uSual. 
Fatigue and Tiredness f 
I found difficulty in getting.up. 
I become f9rgetfi4l, confyaed and un-.^ 
' jnethodicaL 
Feeding Low ajrid. Depressed 
^ tend to neglea myself a lot. 
I have feelings of complete inade- 
quacy. > . ^ 





Children 

Sociability and Family' Retatiohships 
He doesnVwant to know you. 
Me. doesn't want \p mix with other* 
^ children — prefers to stay at home. . 
D^pende^cy 

He wouldn't "allow me to. leave 'the 

room. * 
He. behaves yoiihger. 
Irritable. Moody, and Uncooperative Be- 
havior* 

He's nasty tempered a(nd very hard to 

jplease; * 
He's hard to control. 



of chUdi:en'in the family were noted, and these data were used to 
tabulate the statements {Provided. 4(ather than using individual 
statements, the general categories shown in ubles 2 and 3 were used 
in thpse tabulations^ as to provide adequate numbers in each cell. 
This method evidently provided a very rough indication of the typ)es 
of '^^teriar collected from different respondents. However,.'>it did 
indBcatc'^ hpw much the type of information given varied by the^age 
and sSex of the child. Figui'e 3 indicates very little variation in the 
overall number of caUgories of sutements mentioned for children of 
differing ages— that is, the range of problems did not appear to vary 
by ^gc. A similar finding for the ty^ oF.clinic attended is shpwn in 

■ ' ' . ■■ . ^ . . ■ s , 
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Figure 3. Average number of dimensions in which children experienced 
problems, by age (n = 536) . . 
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EigHre 4. Average number 9f dimensions, in whij 
probjeins, by type of clinic (n = 381) 
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Figure 5. Percentage of children experiencing problems in fach of l6 
dimensions (n = 536) ' " ^ ^ 

Sorrtt categories, however, Vere mentioned much more frequently 
than others: Sleeping and 'eating were the two most frequent?y 
mentioned, while sociability was the category least affected by the 
child's illness (figure 5). Certain categories showed so»^ v;?riation in 
response frequency by the age of the child: fol- exainple, jvpun^r 
children showed more changes in dependency laj^d irrifttbifity than 
older children (Figure .6). / 

Among the patents* reactions to the child's illness there was no 
variaUon iii; the categories of statement fhentioned by the age of the 
child, or by the parity or the^ige of the parent. The only clear trend 
was for parents of higher social class (using the (5ffice of Population' 
Censuses and Surveys [1970] British classification) .to mention syjnp- 
toms of fatigue and tiredness more frequently than. did lower social 
class parents (figure 7). ' 

Tficsc simple analyses give some informatiqil about the possible 
need for different versions of the questionnaire for children ^ 
different ages or for different tyi>cs of condition. , 

In the next stages of the project the . most suitable statements for 
inclusion in a preliminary version of the index will be chosen and thi-s 
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Figure 6. Percentage of children experiencing problems withir) two dimin- 
sions, by age " . . " 
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Figure 7. Percentage of women experiencing problems of fatigue and 
tiredness, by social class (n = 108) 
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will be tested for reliability arnd validity -irv a series of trials. Where a 
. numericj^l assessment of the child's health status is desired, weightings 
representing the severity 6f disability imi^lied by each' stj^ment will 
bt calculate^. In this way the final indek may be used either in a 
simple, uhweighued inanner, tp provide ;^a cortcepiual picture of the 
-problems of a child br a grpup'of chjldrcn or to give a nuhierical 
indication) of the e^te^t of changcvbefore aiid after treatnlent. 

Applications of th^ Index ^ o ' * 

^■^\j^tp:-Sirt atVleast thre^^m^Mi appllic^ suc;h iijdices. First/ 

tn^^rovide an intportarnu rfneans^. fcft: iinprbving 
between the individual patient and - thoi health team,\3y proyidirig a 
, standardized way. for »xfrre5sing th% .'patie^^^ 

they specify accurately ^nd ^Qncisely >ih;e ciefhajnds iHiat he^ h p4aang' 
on th^ health car^^tea^. J^nqufry jirUO/ these (irpblems 'is, of "course^ 
part of good medicaf pra^jgnce, bu|t the index heverthelWs .ri^prcsehts* 
an important timesaving and surift'mSry ctoice^ fon the 'health ^car^' 
professional. K^arty patients Fmdj^ tliat the checkii5t'\ appr^^^ 
>^ inOyes the inhibiti^^ stresses ^spnif tinges expepienced./iri verbally 
" ■' . e)(]pl£lming problenjs to the do<?toi-; , s ; ■ • 

iv/'^,: ^^ is in; ih'e'area .of.-tnedicar 

/'"^"^atiori, since the inforipation'provided by the'se indices pironipte?. 
a.^nprc hol^ approach>to i^atie'ijts''* problems. Although the iriterre- 
■btion^hips^beuvee'n the Individ uali^hcL his social ^nd physical leriviron- 
/ ment; have been systematrciilly e^pWe^^for a number of y^ars, the 
relevance pf individuals* ai^justmients to their life situatioh is still 
ignored in mapf* medical cu^^ula. 
, The third apd perhap^ppst* significant application of the index is 

in the evalujf^ion of car^Rt can b^ emplbyed (and we are testing this 
use) in befofe- and afUK&te studies,' to build profiles of indTividdals 
and. groups, and therelj^o shpW whajL impact i3 being produced Prt 
the quality of life of patimts under 4:reatment, so as to modify , the 
management of the cases acHSe|kj^ 
. , - explored, is the use of the 

;y^^.^^^ i'lifede^ in population surveys to confnbute to the study of needs for 
'^k- «"^rVices. In considering this type o^ application, note that most 
existftig sociomedical indices lack predWtive power and are of 
^relatively low ; value in showing which of alternative strategies should 
* - be adopted in the long term. Thip knowledge of the;increased risk of 
^ disease attributable. Tor example, to age, smpking, or obesity may be 
*used by health practitioners for jjreventive t^re*iind maintenance ofv 
. health, The fiJjAire development of sociotiie^lical indices should 

include tHe incorporation of transitional prob^l^ities (Kaplan et al. 
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197S6) ^tWeen states of socia,! <iysfuncU6ff;^^^^^ ■ 
indication than existsvft present of tht risk|hai iMi!>^nt w fail to ^ 
', recover^ '\' . . ^ ^-'^ . • . ■ 

^ ^ We believe that adding inform^^tioit Cpti<^^^ high nslk factors' in 
•\ childhood, iabout possible asyjpttorn^itic itii^a^^^^ detectable 
; ; ' thi;ough scrieening), and the, acl^a^anloun^ ^ j^^^ to 
vthe basic sociomeditial index wilhimprove i^^^^ The 
index will thenVcombine the av^lablev^ 
■ sutus, rtiof bidity,**-an(l prognosis/ It ish nol^:f^j^^^^^^ hoW much these 
risfc.factor!\such 9$ parity of the. motfe w^ght, or socioeco- 

nomit^statuf\affect functional .j^lgt^^ althouj^ some ^ 

fascinating wO^k has '^en publish^^^^^ of neonatal 

disease' arid loW^birth v^^ghV disturbances, 
^d to I^.ading^attaini1^lqn^^^^^ It may be that 

beljavjoral indices%if the iy^e;|[%ril*^ already reflect 

^ these^bth^r prognose factpr?^ oy^1nt|^^ to explore these 

. ?» V ^possible I'elad^ 8'"show^ a possible model of 

^•^•V'hpVv tneSe variables couj^ irtte^-^rlate. Bfi^ltji 'status (indicated by the 
■ ^^^^"8^ at the ^iffeht) ifs(Jite^a^¥d 16 mbrbidiiy (as measured. 
V. by conycntiohaiK bion^^ functional levels (as measured 

by^p ind^^.Qf th^Xype; and to prognosis (as 

, m^^^d^^ty ihfbrmation OT^^dictf\;^ risk factors). The way in 
s^^^fch^^hese factor«,inte^^ "health" is very inade- 

qu^^y; sflecified, but |he diagraip sJiiSfii^^ that risk factors, such as 
neonatal' 4isease,^r^ and thereby influence 

he-^th status. Alt<g^^ may not produce overt, ' 

* "T^W^I *^rbidit>^;^ infli^rice..a child's functional status in 

^ a#||g.such ^s scholfst^ 

* ^ The impjicatidili^ 'c^^^^^^ strategies of intervention are 

important. I.r^ervention may occur/at four stages: We may intervene 
m modifyittg^^e risk factors,^. often in a program of social action (as 
\<K. ^as showh neces^ry 1^, alt^r t^ mortality rates). Alternatively. 
: ' Strategies ^^'^^^^^E m^y detect presymptomatic disease and, where 
r/ .'^a^^^ itiay improy* health status. Third, the 

^ v.; cpriye^ibnal medical intervention may discoVet^ biomedical aspects of 
:: ; morbidity. FinallVi it i% alsq^'logically possible to intervene directly at 
the level o^ therpati^nt s functi9nal abilities (as is done with the 
provisioq of prostheses),/ ' - 

V. It is cleaiQ|hit cbt^entional medical care represents only one of 
sa^gal pospRe stages for influencing the health status of the 
ij^fMdual: Very plausibly, this is an alternative way'of analyzing the 
results of variations ^n death rates outhned in the introduction, 
^'^^^'y*.^^^'^ ^^^^^ represent the full health status of a ' 

Tpopu^lfon, but they may reflect ^to some extent each of these three 
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EXAMPLES OF DISK FACTORS: 




I ' ;. . -INTERVENTION MAY AIM AT MODIFYING RISK 

/ . FACTORS, MORBIDITY, OR HEALTH STATUS 

Figure 8; Hypolhcdcal model of influence of risk faa^^^ . - A 
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components of rQorbidity, functional levels, and risk factors. We 
conclude that, as mortality rates are too general' to indicate the 
outcoiyes of medical care, a pattern of indices covering aspects of 
risk factors, morbidity, and functional sUttis will provide a more 
specific analysis of:a|^.66tcomes of child health services and of the 
factors which may hfigdify* these outcomes 
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DISCUSSION 

Milton L Roemer and Gabriel Velasquez 

The soctomcdical indicators of health developed and presented by 
Dr. Martini yM|ressed the conference as having great significance' for 
several reaal^^Ulates of mortality and even disability, especially in 
children* arcwRrly influenced by many socioedonomic and environ- 
mental factors, in addition to health chre intervention. By contrast, 
the several indicators of patient feelings and behavior developed by 
Dr. Martini are probably more sensitive reflections of the impact of 
health services. \^ 

In. these days whenAin some circles (e.g., Ivan Ulich and his 
followers), the entire value of njedical care is being questioned — on 
the basis of the n)inor changes in mortality rates associated with 
medical resources — the importance of these more sensitive reflections 
of the effects of medical care on the "qualitV of life'^ is especially 
great. 

A worldwide demand for greater emphasis on primary health care . 
currently exists. Dr. Martini's sociomedicaJ indicators are particularly 
sensitive to the effects of primary care, which are seldom measurable 
in mortality or even clear-cut mbrbidity rates. A high proportion of 
patient complaints — some estimate as much as 50 percent — cannot be 
defined in terms of specific diagnoses. Yet changes in patieht 
behavior or feelings following medical intervention can be docu- 
nienV^ by this technique. 

The method of measuring behavioral changes is also remarkably 
simple and efficient. The patient fills out a questionnaire while he is 
waiting for the doctor, which requires no extra time from either of> 
them. Any changes in bl?havior on a before-and-after b^sis Can be 
readily determined. ' \ 

The exact question lo be, jiosed must be varied with the character- 
istics of the patient (age, sex, occupation, and so on) and the cultural 
settings. Thus the questions on behavidr posed in' England, where the 
technique has been developed, would differ from those to be posed 
in Latin America or in the United States (or in Seattle, Washifigton, 
where the method mgiy soon be tested). Moreover, one must consider 
the differences in answers on behavior or feelings that might be given 
by a child, a parent, or a school ieacher<2f . 

This nniethod of evaluation is npt meant to replace "hard" measure- 
ments such as mortality and morbidity rates. It is intended to 
supplement them, especially with respect to the influence of health 
•services on the quality of life. Such impacts m^y appear, at first 
glance, relevant only in the highly developied and industrialized 
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countries, but they are also rdcvant in* the developing countries of 
Latin America. Many children (as well as adults) in such countries 
obviously h'ave concfitions that do not necessarily result in .4eatb or 
disability, 'but in bchavipral difficulties which can be reduceit ^by 
health vtrvice. „ ** 

Wil;h increasing experience, users of Dr. Martini's questionnaires 
have been aT)le to standardize the questions posed ,to children or 
adults. After repeated trial of certain questions, a range of answers 
has been accumulated which, reflects the various levels of changq in 
the patient's behavior, for better or wprse. Similar standardization 
will be necessary in various national settings. 

Dr. Martini's method, of course, assumes *that the quality of life can 
ha^e a, positive value. It is important tp determine j JIfc^ patient, Who 
has survived a disease, by either prevention or trcS^nt, is living 
and functioning in a more satisfactory and socially effective? manner. 
This appraisal includes* "peace of mind," as well as physical capabili- 
ties. The technique^ therefore, can detect in the patient changes* that 
are not identified by the <fonveriti6nal physical examination. Its. 
application, furtherhiore, can help to make the doctor more awarl^c* 
the sociops'ychological and behavioral objectives of medical care. In 
this sense, this approach can also be useful in medical education. 



APPLICABILFTY OF THt TRACER lOiETHOD TO CHILD 

HEALTHCARE 

•I 4 • . 



David M. Kessner ^ 

A, . 

^/ , SUMMARY 

• This report describes the results of a field study designed to assess 
the health status of a defined population of children representing a 
range of socioeconomic characteristics and to compare the care 
delivered to these childfen by different types of medical organiza- 
tions. \ set of tracer conditions was ^elected, and process ando 
outcome data were analyzed by pertinent demographic, social, and 
mj^dical care variables. - , 

The major conclusions bf the study ire that prfeFvalence rates in the 
community were extremely high*for all tracer conditions; socioeco- 
nomic status of the children influenced ^he pi;evalence rates of some, 
but not all, of <he tracer conditipns; and prevalence r^tes for the 
tracer coaditions did not vary significantly by provider organization 

when children of similar social statrfs were compared. 

' ' ' * , - . " 

'.^ Xhe limitations of this methodology and field study are discussed. 
^>eneral quality assurance methods ^re considered in lig^ht of their 

cost, utility, and potential impact on the health of patients. 



^ .PURPOSE OF STUDY . ' 

. ■"' ■ . ^ 

^ In 1969. Board on Medicine, later to become the Institute of 
Medicine of the National Academy of Sciences, began a program to > 
study and compare the health services- received by contrasting isjitial 
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and economic jffpulations. The commiUee responsible for this pro*- 
-• gram" chose to evaluate the null hypothesis**that there are no 
differences in specific measures of health status in children of 
comparabl*^ social and economic status, regardle^ of the source of 
their uAal ambul^ory fn^dical care, 

The findings- djscussed in this paper have'^en reported in a 
monograph entitled A^ssment of Medical Care for ChUdrer^orUrasts m 
Health SkUusy Volume 3. publbhcd by the Institute of Medicine. National 
Academy of Sciences. 1974: 

A large interdisciplinary staff participated in the various phases of 
this prdgram; without their efforts this report would not have been 
possible. Carolyn Snow dnd James Singer, my coauthors on volume 5, 
are responsible for much of the data discdssed irT this presentation. 

To explore the question raised by the cori\mittee. it was necessary 
to (1) define different organizations for the delivery of ambulatory 
health servfces; (2) develop a methodology for studying the qualify of- 
healtfi services furnished by these different health providers; (3) 
identify populations that represented a spectrum of sofial and 
economic characteristics, and that used a variety <5f ambulatory care 
facilities, and (4) undertake a field progranj'that wpuld ^Wow the 
evaluation methodology to be tested. 'the results to be subjected to 
crjtical review and. if warranted.' the methodology to be replicated by 
other investigatory on different populations, 

. The National Academy of Sciences is a nonprofit, private organi- 
zation, founded under a congressional charter to act as science 
adviser to the executive and legislative branchc;s of. the United States 
Government. Thiy study was supported by a 'consortium ^of private 
foundations and gbvernBfient agencies.^ — 1 



» Health Services Research Panel: S.M. Nabri^, Chairman. The Souihern, Fellowship 
Fund: K.W. Deuschle, Mount Sinai School of Medicine; A. Donabedian. University of 
Michigan School of Public Healtli; LJ. Duhl, .University of California. Berkeley; R. 
Fein. Harvard Medical SchootI: C.L. Hudson. The Cleveland Clinic; R.C. Long, 
" LoubWHc. Kentucky; D. Mechanic. University of Wisconsin; C. Phillips. Cook County 
Hospital. Chicago, Illinois; N. Piorc. Columbia University; E.W. Saward. University of 
Rochester School of Mcdkinc and Dentistry; J*B. Turner. Case Western Reserve 
University; A. Yarmolensky. University of Massachusetts; A.S. Yerby, Harvard School* 
of Public Health. . * 

• Major support fOr this study was provided by the Carnegie Corporation of New 
York. The wprk was also supported by the Fannie E. Ripple Foundation; Association 
for the Aid of Crii^ed Childrenj John Hancq^k Life Insurance Company; Office of 
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AREA OF CHILD HEAtTH CARE 

■ . " . ' ' ' • 

The basic methodology centered *on,ihe use of specific- indicator 
health problejns called tracer t^nditidrfs. The tracer concept was. 
based' on the premise thai carefully selected diagnoses, prpbjems, 
symptom compfcxts, drugs, "or procedures, when combined in sets^ 
can provide n»nageable data 'to exarqine both^the process and 
outcome of meoicat .care. These specific ^ind icators enable health care 
evalu'ators t6 assess the health sjtalus of if defmed community 
population, tjp evaluate the prQC*es;s and outcome of care delivered by ^ 
different 'pr^tice settings, and to begin to consider Jthft^ interaction 
need.ed "Shimon g a-frommupity, its.pcople,'»and thie proVili^i;s .of health *^ 

. servces in order to imj^rpVe the health jof -chil^ren.^ , • * 
Before thfi field study began, criteria were develoj^d for selecting 
tracers. .Aiv it^itial set of tracers was designated, And explicit medical 

. management criteria for. t,he individual tracer con^litions w^re speci- 

; fled,; Irf addition,, the -^epid^ntiplbgy and functional impact of eac^ ', 
tiracefcdndinon were»Jra^marized. ' , ' . 

this field study^lPUet of tracer conditions that included the 

-/ following common p>eai*at'ricf.prohleips was investigated: middle ear 
infection, conductive hearing^ loss^ iron-deficiency anemia, and visual 
disorders. Table 1 shows t-hiit each of the f^^t^e, ipatpgories of medical 
car© activities' presented- — pirevention, screening or case finding, 
evaluation, management, apd follow-up — is highlighted by at least 
two of the tracer, condition^. Health counseling and prope'r treatment 

■ . r • , ■/ . 

Table,!. Aspects ol the Process of Pi imary Am,bulator)t Health Ca^e' 
Highli Jilted by Accepted Tracer Conditions ' ' '* 

^ ■ Triicer*C".onflitions 

■ . . — ^ — ^ 

Miclcile^ . Hpaf- ^j^^n Iron deficiency 

eflV^ ing ^ ''^ anem^., i 

. Prjxc55 Activities^ ' itit'cctiot\ ,^ ■ loss ' 

Prevention. . . -H* . ' ' .^MJ^ * \+ - ^ 

Screening ^ • * + +J ^ 



Evaluation 

History ^nd , ^ , 
physical exa'jftjv * + * ■ » ■ 

■ Uborutbry -.^ib^ - i " • - " , ' ■ ' + 

Other testing v; ,*, . ^ ^^T* + ' 

Management ** ^ V ' . i * \ 

Chemotherapy . - + ' / + 

Health counseling ' ' ■ : + , * . - + 

Specialty referral + + . + 

Hospitalization . * . ■ , 

bllowup " ' . + \- V . 



0' 
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of middle ear infections can be effective in the' prevention of^ron- 
deficiency anemia, and follow-up is often essential for acceptable care 
pf children with middle ear infection, conductive hearing loss, and 
iron-deficiency anemia.. , 

.To use these conditions as measures of the health status of children 
and as indicators of the way physicians practice medicine, it was 
necessary to specify diagnostic and therapeutic management criteria. 
. The criteria could be used for measuring the prevalence pf the 
condition in a defined population and as standards for assessing the 
performance ^of a medical care organization. These criteria repre- 
sented a mirtimal care managejoient plan that a group of practicing 
famHy physicians and sp^ciatfsu^felt wbuld be useful guidelines for 
evaluaung ambulatory health services in a population with these 
conditions. The Jpt-iteria were 'not proposed as a rigid treatment 
formufaj Applicable to individual patients. 



< 




GENERAL METHODOLOGY 



A major goal of this study was to assess the ambulatory health 
services cjelivered by different types 6{ medical care Organizations. 
The study was intended to encompass persons of different income 
levels, defiiie their usual source of ambulatory care (including those 
who did not identify with any specific provider), a^d compare- 
within and among different social classes— the performance of these 
organizations and the health status of the children who received care 
from these providers. These objectives required that the families and 
children participating in the study be representative of the commu- 
*nity, rather than the particular patients of specific medical care 
organizations. 

Logistically» the study was divided into four separate but related 
daW collection efforts. First, a probability sample of families residing 
in two areas of Washington. D.C,,,,aqd a random sample of enrollees 
of a neighborhood health center (NHC) were selected. Interviewers 
gathered data from tK^ mothers concerning heahh attitudes and 
orientations, medical care utilization, problems in the receipt of* 
health care/ s9cial and economic position, social participation, ani^ 
tracer-specific medical histories of their)young children. 

Second, the children.^"\^ 6 monmsthrough \\ years, whose . 
families were included in th^'^^dy, were evaluated clinically for 
specific tracer conditions— middle/ ear infectipn, conductive^hearing 
loss, visual disorders. aVid iron-djeficiency anemia. Historical, physical, 
and laboratory data were gathered independently to provide clirtical 
measures of the children's health status and to provide data for • 
outcome analyses. 
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Third, providers identified in the household survjjy as the "usual 
source of care" for the children were sur^^eyed. Data were collected 
on .the professional background and activities 6f the physidanfe, the 
organizatibwal structure of their practices, and selected aspejctji of 
their usual ^process of medical care. ^ ^ 

Fourth, medical records- were abstracted to record the medical care, 
delivered to a sample of the children for- the tracer conditions and* 

selecftd outcome measufes*of that care. ' - 

• • . ■ 

i^AMPLE DESIGN 

Community Sample /rv _ y 

The sample population was restricted i6 families wtth at least one 
child 'age 6 months through I J years who reside^ in hou.sehieids in 

^two contrasting socioeconomic ireighborhoods iiv Washington, .D.C. 
Tl^^Shephecd Park-Takoma i'rea-compriscd 10 census tracts in the 
northwest part' of the city that included a majority of middle- to - 
upper-income families,. Xh't .seven censirs tiacts that constitute the 
Congress Heights area in the southeast part uf the ciiy are made up 
predominantly of lower-income^families. 

To ensure that the final sample would include a sufficient number 
of children using different medical care organisations, the study 
population was -selected in a t>vo-staged sampling process. In the first 
stage, a random sample of all residential addresses in the two- 
neighborhoods were surveyed to identify families with children ag^ 
6 months through 1 1 years and to a.scertain the children's usual 
sources of care. In the second stage, families with children in this age 
range were sampled at difflbrent sampling ratios, depending on their 

^ usual source of care. All families whose children used the^less 
common source of care — pncpaid group practice — were sampled, 
while only half of the faiifulies ^ho took their children to more 
common providers, sych as' solo 'physicians, were included in the 

^study. Figure 1 shows the various sampling stages, the number of 
families and children included at each stage, and thi^ final number of 
fahiilies and children that are the subjects op^his study. 

Households outside the survey boundaries, ntinre.sidential proper- 
ties, and families with no eligible children were excluded from the 
one-in-four sample of residential addre.sses. The overall completion 
rate for the screening survey— the first stage of the sagiple — was 91.0 
percent. A, tojal of 2,9G[p families had children in -ti^e designated age 
range whose usual source^ of care could he identified. An additional 
sample of 711 families, tor whom no idcn^fi^ion' was available, on 
the family compx)sition by age. was included. 



Figure 1 . Sirttpl^SiA sampling scheme for ^election of the commuifi 
selwited geographic areas in Washington, D.G., 1971. Of the tota, 
of famiUe^ eligiUlity by children's ages coqld not be determined 

' ^miUcfl. (from fessner et al; (1974). Reprodua/d the permi 
the Natipkial AcadeijBy of Sciences]. ^ w f 

All families who r^rted taking their QhUdfen to a prtpaid groun 
practifx or^a fe^for-service group or reported having no paiticulaiX 
source of, reguSr care Were included ihahe sample. They were 
sampled, therefore, at the rate of on^n four of th<? total population. 
.The families who reported taking their chUdren to- publifc clinics, 
pkivate indivildual/or partnership practices, and KospitaLio^tpatient 
departmentybr emergency rooms were subsampled afthe ra^^one 
mtwo. or fae in eight from the total ptopalaUon. the final eligible 
arid interview sample ilcjuded 1.436 families dnd 2.780 children. 

Neither of the d^ed geograpliic artas sampled for the commu- 
mty population w^ served by an Office of Economic Opportunity 
NH€. To include ah NHC among the provider organizations, a 
supplemental sample of famOies using the NHC located in' the inner 
dty Cardoio afea of the District of Columbia was added to the study; 
Families with at lea^t one child age 6 months through 11 years were 
selected random|y fipm the enrollment re<cbrds of the Cardozo NHC 
A total of 775 eligible families was identified.'Systematic sampling of 
this grcAip at a ratio of 2 in 3 resulted in a population of 516 families " 
wh&lmn^ initiaUy contacted. Of this group, approximately on^ourth 
coukl not be located. Afradditional oneeightK were located bb^could 
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not be contacted, haid left the area, or were de< 
refused to participate. The final eligible and cdloperatin^ 
included 289 families— $6 percent of the origin;|l sam 

HOUSEHOLD SURVEY QUESTIO; 

, The ^household questio^jl^iair.e was designed to gat 
^ kinds of infbrmation< 1 

— Me^i^l care utilization, co¥ts^and satisfaction; 
— Barriers tQ receipt of medicJaf care; ; . v. 

— Dcmogiraphic a ndsocioeconomic characteristics; 
' — Health knqwledgeland attitudes; ' 
— Perceived health status;. 
— General attitudes and lifestyles; 

—Tracer-specific medical history (eligible Children ;pnly), , 

All family information was obtained From the mothe/ or mother 
surrogated In hpuseholds where more than one family with eligible 
childrep resided, supplemental interviews were conducted wi\h each 
V mother. The completion rates for the household field wotk are 
presejited in tabl^ 2: Nonresponse rates varied from lJCr37to 14.5 

Tabic 2. Househ^j^mnterview Completion Status for Shepherd Park- 
Takoma and Congress Heights V ^ - 



Number 



Percem 



Complete interviews 

Noninterviews 
Refusals 
Noi at'home 

Subtotal 

'indeterminate eligibility status 
"Moved, could not be locate^ 
Other 

Subtotal ^ 

Nonjample cj^es 
No eligible children 
Moved outside the area 
House vacant ^ 
Np such address , 

I * Total 



58 ' 
13 



309 
43 
12 
5 



1.436 

173 
1.609 



71 
1.68Q 



369 
2.049^ 



89.3« 

(85.5)*» V 



10.8" 

(10.3)*» 



4.2^ 



18.0*^ 



t Based on total ^mple of 1.609 eiigibie ;respondcnts. « 
* Based on total sample of L6S0 eligible respondents. 
' Based on to^l of 2.049 sample and nonsamf^e cases. 

' Includes 62 families added from households where more than one family with eligible children 
resided. _ . ^ 
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pci^ent In the NHC sample, 100 percent of the eligible riespondents 
vltob donscntcd to participate were interviewed. : * 



• • - CLINICAL SIJRVEY 

'Methodology * ^ 

• All i^hildren ages 6 months through 11 years in the Tamilies* 
interviewed were eligible for clinical/ examinations. QfjCKe^others 
interviewed, 90 percent gave permission for ^their children to be 
examined. Because age-specific estimated prevalence rates varied for 
each tracer condition, and because it is difficuk to perform, sophisti- 
. cated sensory examinations on -prescKbol children, different^ sets of 
tracers wcff^trhosen for the two age groups: ^ ^ * 

Age G\ oup pr Tracer Condition 

6 months through 1 1 years Anemia 

■ ■ , ' t Middle ear infection 

4 months through 1 1 years Middle ear infection 

. ^ V Hearing loss 

* J Visual disorders 

Tb control for seasonal effects on Hhe inc;dence ^jof middle ear 
infection, all clinical examination1]^re completedybet^e^n January 
and April 19X1— the^jB-tnonth period o£ highest incidence of the 
acute conditioh. : ■ > • 

Protocols for ear, rtose, and. throat, examination; audiometric and ^ 
Mteib^ tests; and hematocrit determinations were developed with the 
. assistance of consultants. A. comprehensive otola^ngologic examinjf 
tion was^pcV^rmed b^p. board-cer^fied or board-eligible otolaryn- 
J ; .gologists. Any a^orn[iali^i>!:^ the ear examination other than occlu- • 
^" sion of th€L auditory canaPwas considered *a' criterion for failure*: 
Children agfes 6 months' through 3 years^who failed the otologic 
* * examination,rolus a 10-pereent random sample df those who passed, 
■ had impcdaiKe testing performed. Children i^ges 4 through 11 with* 
* Otologic failures had pure tone hearing -threshold and impedance 
tests. OldeiNchildren wfio passed the jentologic examination were tested ^ 
v^lth ^pure tone hearing screen. Those who failjed this screening test, 
. jiiilVs*a 10-porcent random sample of those \vho passed, were also 
'given the hearing threshold and imp^ance tests as shown in figure 

2,;. \- . ^ , 




MIDDLE-EAR 
IMPEDANCE 



I • 



HEARING SCREEN 



* Figure 2. Flow of children through clinical elimination stations, [Frorp Ressner et al, 11974). Reproduced with the 
permission of the National Academy of Sciences] 
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X^linical Criteria 

Dynamic impedance measurements of the middle ear provide an 
objective assessment of middle ear functioning and have shown high 
sensitivity to subtle changes in the integrity of'the middle eai 
network. The fail criterion for this field study was a rjegative middle 
ear'pressure equal to or greater than 100 mm H^O, 

Pure tone hiring tests (screening and thres*hold measurej^ents) 
were carried out by the staff of the Children's Hearing and Jpeech 
Center, Children's- Hospital National Medical Center, 'The pure tone 
hearing screening test was done at test frequencies of 250, 500. 1000. 
2000, 4000. and 8000 Hz. The , screening level chosen for the stady 
was 15dB (ISO. 1964). The fail- criterion was no response to two or 
more test^tones in either ear. 

Pure ton* air conduction thresholds were obtained for test tones 
from 125 through 8000 F^z. and bone conduction was assessed from 
250 through 4000 Hz. Failure criteria for, these test? includeS-^arTW- 
conduction threshold of 20d'B (ISb.T964) or greater in either earal 
any test frequency and/or a 20dB air-bone gap at any frequency \ 
regardless of ear. * > 

The vision* screening examination for acuity, organic abnormalities, 
pculaf motility, aftd fusion Was performed by second-year ophthalmic 
assistant students.- Discrete pass-fail criteria -were established by 
consultant ophth^mologists. and all children who failed the screening 
test, plus* a lO-percent random sample of th,ose who passeck were 
given diagnostic opjithalmologic exaniinations with cyclqJfcia'by 
board -certified or board-eligible ophthalmologists. 
P vFingertip blood specimens were obtained for the microhematocrit 
test. All specimens were processed 'by the hematology laboratory of 
Childi:en's Hospital. A hei:natocrit of ^ess than 34 percent for those 
children younger than 19 months, or less than 35 percent for those 
children^older than 19 months, was considered a failure. ^ 

' FOLLOW-UP CARE ^ . 

^ Part of our obligation to those who participated in the study was to 
provide treatment for children whose families had -limited financial 
resources and no regular source of nledical care. Nonsurgical care 
was provided fq^r acute and chronic otolaryngologic problems. Pre- 
scriptions were given for glasses, and^'^'ill children with motility 
problems were refeffed to t»he ophthtitfinology clinic at Children's 
Hospital. / , jf ^ 

The resixKs of all tests were sent to |jj|rents and/or their children's 
physiciaQj^^pon parent request. U^S^^ 
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PROVipER,SUBVEY 

The final phase of this study focused on the physicians who were 
the usual or primary source of care for the community sample of 
children (those residing m ^e Shepherd Park-Takoma and Congress 
Heights areas) as .well as those children enrolled at the Cardozo 
NHC. Information obtairied from the providers included the follow- 

— Characteristics of the physicians, including medical, training^ satis- 
faction with patient and practice situation, and atUtudes toward 
health care organization al an d financing issues; 

— Characteristics of the organTzirtw^nal setting of the practice includ- 
ing facility, nufnber aVid types* of personnel, accessibility, and 
charges; , 

— Medi(;al process? data from the medical records of a sample of 
children who did and did not have a history or clinicaKeyidence of 
tracer-specific pathology! jw ' ^ ' ♦ - 

The provider studf was divided into two stages. In stage one, data 
were gathered about physicians and their organizations; in stage two, 
data were abstracted from medical records. 

Survey of Organizations ^nd Physicians v 
., «► - , 
In' stage one, data were collected witK a self-administere^l question-, 
naire sent to all jprovidiers who had been identified as the primary 
and/of usual source of care for the .total sample of children. This 
information was supplemfented by data from the physician master file 
maintained by the American Medical Association's Physicians* Records 
Service. * , 

Sample ' -I * . 

Criteria for *'usual provider** were based on information given by 
the mother during the household survey. All practices mentionecj as 
the usual source of care were contacted. If the usual provider were a 
physician -practicing in partnership, all physicians in the practice were 
included in the provider study. For hospital -outpatient departments 
or emergency rooms and public and private clinics, all staff physicians 
who provided care to children were included. ^ • 

The total sample for stage one included 178 provider organizations 
and 312 individual physicians* The organizations and physicians were 
categorized into six predominant medical care organizations as shown 
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s 

, Tabic H, Distribution of Provider Organizations and Individual Physicians 
by Child's XJsual Source of Care, WashinRton; D C., 1972. 













Neigh- 


1 








Prepaid 


Assoc./ 


borho^ 




S6lo 


Hospital 


Public 


Group 


Private 


Health 




Practice 


OPD/ER 


Clinic 


Practice 


Clinic 


Center 


Total 


Provider organizations 














Number 1 10 


12 


16 






1 


178 


Percent 61.8^ 


N 6.7 


9.0 


1.7 


20.2 


0.6 


100.0 


Individual physicians > 












Number ^ 110 


81 


25 


14 


' 79 


3 


312 


Percent 35.3 


26.0 


8.0 


4.5 


' 25.3 


1.0 


100.0 



in tabic 3. Organization quesjjonnaires were completed by approxi- 
mately 85 percent, and physician data were obtained from 80 percent 
of the providers. 

Survey of Medical Records 

Stage two consisted oPabstracting the dutpatient medical records of 
a sample of children. The chart review had three objectives: to 
validate the utilization , data obtained from the household interview; 
to describe the records system used by different medical care 
organizations; and to assess the process and outcome of care given 
for the tracer conditions to a sample of the children. 

Sample 

The sample was limited to the medical records of children whose 
only source of care was a hospital outpatient department or emer- 
gency room, the prepaid group practice, or the NHC. A subsample 
was selected from all eligible children whose parents gave permission " 
to have their nf\edical records reviewed. All children who were 
identified as using more than one provider type as the usual source 
of care were excluded from the sample. The remaining eligible 
children were then stratified by the . fdllowing variables: type of 
practice, area of residence, and presence or absence of tracer-specific . 
pathology. The final sample included two thirds of the users of the \ 
NHC, two thirds of the hospital outpatient department and emer- 
gency room users, and all of the eligible children enrolled in the 
prepaid group practice. ' 

Overall, 77 percent of the sample medical records were located, 
ranginj; from 97 percent in the prepaid group prAetice records to 
approximately 60 percent of records' in the hospital outpatient 
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departments. Of the located records, 92 percent contained informa-. 
tion pertinent to the time preceding our clinical examination of the 
child. J 

.» , 
; RESULTS • \' 

^ . . 

Examples of results given herein are limited to. the major commu- 
nity^ sample. The data for the sample of NHC users are not. 
■ considered. 



ill' 



Characteristics 'df thp Families and Children 



Approximately 94 percent of the 1,436 families in the final 
community sample wert black. The explanation for the very high 
proportion of blacks in the community sample (census data for thes 
two areas reported an 85 percent black population in 1970) may be 
our requirement that each eligible family included in the- study have 
at least one child between ages 6 months and 1 1 years. The mean age ^ 
of the mothers (respondents) was 31 years, with 3 percent under age 
20, 45 percent between ages 20 and 29, and 16 percent age 40 or 
older. One third of these families were headed by separated, 
divorced, widowed, or single females. A broad range of .family 
income was represented by the community sample. Approximately 
one fourth of the families had annual incomes of less than $5,000, 
one fifth had annual incomes of $10,000 to $15,000, and almost 15 
percent had family incomes of more than $15,000. Analyses oji the 
respondents* educational attainment and occupation were generally 
consistent with the family income data. 

The distribution of. the source of care received bV the 2,780 
children from the community sample is presented in table 4. By 
individual types pf provider organization, almost 30 percent of the 
^•fc ■ 

Table 4. Distribution of Eligible Children by Usual Source of Medical Car^: 
Community Sample. Selected Areas in Washington, D.C., 1971. 



Usual Source of Care 


£ligit)le Children 
(% distribution) 


Solo physician 


29,5 


Hospital OPD/ER 


2^.7 


Public clinic 


15.8 


Prepaid group practice 


6.3 ' 


Association, private clinic 


10.4 


No source 


3.3 


Multiple provider types 


6.0 


Total 


100.0 


Total number (all sources of care) 


2.780 



lb 
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children received care fronTsolo practitioners and an additional 10 
percent from physicians in small associations or in fee-for-service 
clinics; only 6.3 percent were children enrolled in the prepaid gVoup 
practice. Slightly less than 30 f^rcent identified a hospital outpatient 
department or emergency room as their source of care, and 16 
percent used public clinic!|. Finally, children with no identified source 
of care accounted for 3.3 percent of the population, and 6 percent of 
children received care from multiple types df organizations. 

Health Status and Tracers * . 

A primary objective of this program was to use specific health 
problems as a means of contrasting, the health status of aidiver^ 
group of children, ages 6 months to 11 years. The health status of 
the children was assessed by independent clinical examinations 
carried out by a team of otolaryngologists, audiologists. and ophthal- 
mologists. All children received standardized tests to determine the 
presence or absence of the tracer health problems. The criteria for 
deciding whether a child had a particular condition were set prior to 
the study and were applied uniformly to all of the children. 

Selected examples of measures of the children's health status and 
provider process and outcome data from t^e children in the commu- 
nity sample are* presented in the following sections. 

Anemia 

Iron-deficiency ^nemia was one of two indicators of the health 
.^atus oi children kges 6 months through 3 years. Because normal 
hematocrit values vary with age, our criteria for determining anemia 
were based on age-specific hematocrit values. A child was categorized 
as anemic-if the hematocrit level was less than^4 percent for chrldrep- 
younger than 19 months, and less than 35 percent for children 19 
months or older. . 

Of the 505 children tested. 26.3 percent were classified as anemic. 
Most of the anemias detected were mild. Hematocrit values ranged 
from 26 to 42 percent in those younger than 19 months and from 28 
to 49 percent in the older group. 

There was no statistically significant difference- by age, sex, or .color 
in the proportion judged anemic. There was, 'however, a strong 
association between anemia knd the educational attainment of 

the child's mother (figure 3). Prevalence rates vaned from less than 
19 percent in childr^ whose mothers ha* education pas,t the high 
"School level to almost 33 percent among those whose mothers had 1 
to 11 years of schooling. Analysis shows the relationship between-the 



Fifiiire S. Propoitioii of children ages 6 months through S years with anemia, 
by education of mppndent: commuaity sample, selected areas in Washing- 

- ton, D.Cm 197^ Total chi square (2 df) - 7.982, ><0.025{ renxsssion 
within chi square (I dO " .7.611,/<0.0I. tFrom Kessner et air(1974). 
Reproduced with the permission of the National Academy of Sciences] 

mother^s education and the anemia fail rate to be inverse, linear, and 
^statitticaOy significant. 

The relationships between the preva^ehce of anemia and the 
children's usual source of care is cfearly influenced by the«a^sociation 
of tbS^'iBpt^er's education and anemia, and the known differences in 
therimother^s educational level among different provider organiza- 
tions. Thus» t^ie^CTude anemia rates by usuadjspurce of care varied 
from Just ufffoer 18 percent for those who regularly went to 
associations of physidans to almost 32 percent (figure 4) for diil^ren 



1 I 




Bgure 4. Proportkoft of children ages 6 months through 3 years with anemia, 
by usual sources of care: community sample, selec^d areas in Washington* 
D.C., 1971. To^al chi sqiuire (5 df) - 3.945, /»>0.5. [From Ressner et al. 
(1974). Rejprodubed the permission of the National Academy of. 
Sciences] • ^ . 
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Tabic 5,. Prevalence Rates of ^n^ia Standardized by Respondent's Educa- 
tional Attainment Within Provider Group: Community Sample, Selected 
i^reas in Washington, D.C:. 197 L * " . ^ ^ . 



Usual Source of Care 



Anemia 



•f Assoc./ 
Privalc 



Solo 



-No 

Prepaid Source/ 
Group Public Hospital Multiple 
Practice^ Clinic OPD/ER Source 



Total numWr examined |37 53 J21 97 

Observed cases 8 33' 12 .33 30 

ExpectW-cases 11.3 31.6 12.3 34.3 26.9 

Observed to expected ratio 0.71* 1.05 0.98 0.96 1.11 

Chi square (1 tlO* • 1.021 0.052 0.Q04 o!o42 0^441 



• Total chi square { 5 dO * 1 .965, p>0. 



T 



48 
13 

J2.5 
1.04 
0.000 



using hospital outpatient dqjartments and emergency rooms. How- 
ever, when an indirect staitdardization methpd was used to adjust the 
anemia rates for differences in the mother's educational attainment, 
virtually all variations in the prevalence of anemia among the 
providers were eliminated (table 5). These data show that the ratio of 
observed-to-expected cases for each usual source of care approxi- 
mates that found when the educatiori of the mothers within each 
provider organization is considered — that is, essential differences 
between the number of observed and expected cases do not exist. 

Medical .process and outcome data were obtained by reviewing a 
sample of patient records. As shown in figure 5, of 169 children ages 




Figure 5. Disposition of a population of children ages 6 months through 3 
years based oir record review for anemia: neighborhood health center, 
hospital outpatienV department, and prepaid group practice samples, 
Washington, D.C.. 1972. [From Ressner et al. (1974). Reproduced with the 
permission of the National Academy of Sciences] ^ 
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6 months through 3 years, slighdy more »than 70 percent were 
screened fo ranemia. Of those children with a hematocrit or hemoglo- 
bin test recorded at any time in their charts. 36,4 percent had anemia, 
based on the criteria used in this study. Almost two thirds of the 
screened children v^hose laboratory tests indicated th^ were anemic 
had not been diagnosed or treated for the problem. Only 16 of the 
44 children identified as .being anemic had been appropriately 
diagnosed and/or treated with iron. Because so few children were 
actually treated for anemia, no outcome assessments* could be made 
on a provider-speoific basis, , 

• * 

Visual Disorders 

' * Uncorrected visual disorders constituted one of three healtlj status 
measurps used for children ages 4 to 1 1 years, A comprehensive 
vi«on screening test was gi^^en to 1 ,670 children. Pass-fail categories 
were established for each of the three components of the screening 
test — acuity, ocular motility, and organic disorders. Children who 
failed one. or mort of the three components of the screening test 
were considered to have failed a summary category. Because the^ 
component measures are not mutually exclusive, the sum the 
proportions failing each component ^dd to more than the totalTailing 
the summary criteria. 

According to the summary criteria, 26,3 percent had iincorrecteil 
or inadequately corrected visual disorders, 19,1 percent had pQ^r 
acuity, 4.6 percent had ocular motility disorders, and 6,6 percent hiid 
organic diseases. The proportion of children failing the visual acuity 
and organic disease categories Was related to age. There were no sex- 
related differences in fail rates, nor were differences in fail rates 
noted between black and white children. No consistent asst>ciations 
were found between pass-fail rates for the component or suWfmary 
vision measures and a variety of .social class indicators; including the 
respondent's educational attainment. 

Figure 6^demonstrates clearly that there are no significant differ- ' 
ences in the proportion of children ages 4 to 1 1 wKo failed the vision 
screening summary examination by the usual source of their primary 
medical care. Indeed, th^ striking feature of these data is the narrow 
range of fail rates — from a low of 24.3 percent in the children using 
associations, of physicians to a high of 28.9 percent of children using 
public clinics as thei^Jjsual soured of care. Children with no regular 
source or with multiple j^Surces of care had a fail rate of 27.6 percent, 
only slightly higher thg^ the overall f^il rate of 26.6 percent. These 
data certainly do not indicate that the proportion of children failing* 
this comprehensive vision screening test was related to whether or 



Figure 6. Proportion of children ages 4 to il failing vision screening test — 
/summary criteria-^y usual sources of care: v community sample, selected 
areas in Washington, D.C.. 197.1. Total chi square (5 df) « 4.075,>>0.5./ 
[From Ressner et sd. (1974). Reproduced with the permission ' of the 
' National Academy of Sdenees] 

not they ha^ a lisual souVce of primary ambulatory care or to the 

• naiture of the organization that provided the care. 

The physkian questionnaires revealed that high pipportions of: 
physicians from some medical care organizations claini to screen 
routindy for visual disorder»H90 percent of physicians in prepaid 
group practices and 50 pcircent of those in . hospital-based praittices. 
However, review of 489 padent charts did not conflrm the physiciansV/ 

^claims; four fiftl^ of the children studied did not have a recorded 
visual exam. The propottioh of children without vision- screening 

>in^nnatiQn on' their charts varied from 60 percent in children who 
received care from the prepaid group practice to 95 percent in 
children receiving care from the hospital*lxued facilities. 

Aihpng'the children screened, one third had an organic disorder 

. or an abnbrmality in visual acuity or mpdlity. Although we did not 
attempt to assess directly the adequacy of the specialty care for 
children with visual disdrders, data from our clinical evaluations 
pirovlde inforrilation on the adequacy of correction in children who 
wore glasses. 

Overall, 7.5 percent of all the children ages 4 to 11 years tested in 
our community sample wore glasses. As expected, the proportion 
varied by age and the education of the mother. The 4idequacy of and 
need for glasses were assessed by comparing acuity test results with 
and without correction for children who wore glasses. As shown in 
figure 7, the adequacy of correction was unrelated to the mother's 
. education. A striking 72.4 percent of 'children wearing glasses were 
not corrected appropriately or adequately — these children passed the 



in 




Rgute T/ftopordon of chi^ gbMes wboie comction 

WM inappropriate or inadequate, by education of respondent: community , 
Mfl^ selected areas in Washington. D.C.. 1971. Total chi square (2 d£) - 
0.789. p>l; regraAkh mthin^chi square (1 Vf) - 0.7l9m p>0.1i [From 
iMssaer et al. tl974)ir«:9eproduced with the permission of the/Iational ' 



Academy of Sdences] 



withoitt their glasses or faUed. both with and . without glasses. 
Itestms niniter tbViese have been reported by Safir and his coUeagueS 
( 197S) in a study of diQdren in New York City. ' 

UKOT AT1(3NS OF THE 1^ 

Thm are/no pitiUei^^ evaluating the quality of 

health services. AU approaches have limitations and are often plagued 
by Qiethodologic and/or analytic weaknesses; 

In considering th\ broad types of data Used* w^khbw that the 
asstiisnic^t of stnicti*e reKes on the assumption that the quality of 
the facilities and the qualifications of the health professionaTs are 
related to the process and to the outcome of medical care^a doubtful 
aasuniptidn tm most instances (Payne et sQ.* 1976): Process evaluadon, 
based on predeteniA^ned or judgmental criteria, is knowii lo produce 
resulu that differ from outcome studies (Brppk 1973). Indeed, in 
some instances, a poor process has been associated with a good 
medical outcome. These weal^esses of structure and process evalua- 
tioi^aye led many individuals^researchers, administrators, ind 
po|ftidat)s-*-to encourage reliance on outcome studies alone. This 
advkie makes 'sense because, ostensibly, outcome evaluation direcdy 
measures a person's "health" or factors related to it: 

On closer scrutiny, however^ it.becomes clear that outcome studies 
often cannot be performed unless criteria for process evaluation are 
also esubliriMd: First we must define the population with the 
problems ,wbp|se outcome wil} be studiedL. Further, the intrinsic 
limitations of outcome evaluation; including proximate or jhort term 
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outcomes, are significant. Frequently, "the pFje^lence'qf the ouubdme 
) to bt*studied^is very low; the outcome ;tinay be sigriificami^i>fltrcMed^ 
by >;ariables thjat-are beyond. 4he contrpJ of the medical^paf^ systerTv::=»^ 
ge;pctit:, -ftociaf/ cultural, and economic factors. In addition^ there^ is 
t)ften difficulty^'in obtaining^utabme iTTfpririatio'p without fiollowu^a 
patientjjinterWws ^nd f^ts — logistically "complex and exp^^fffe pro- 
^ cedufes to, carry out, [ ^ / 

V In 'a recent comprehensive study of ©4/fcome method, ^ro6k 

^ and his colleagues ( 1^76) sumjjgarized the statr of the art in outcome 
evaluation as follows: ^ '..1^ i»- 

^ - I'.th^frt exists a pMradQxical situation, ifi y^^j^ poliCyt demands that 
■ '-^ operational quality assurance systems use tnl outcome method to 
assess quality carej» wjfiiie there is « dearth of valid and reliable j 
"^^utcome criteria" and st^nd^rds and no mefhod of proven feasibility* . " 
by which they can be applied. ^ \ 



As ihere are serious littiitations with the types of d^ta used in 
evaluation studies, so rhere are with the general methodologi'c 
approaches. The tracer method, as initia^y desc^ribed, was based on 
the premise that one can carefully^ examine selected medical care 
activities and their outcomes to gain insight into the quality of general 
medical care that is being received by patients. The results of this 
study ("kesmer et al. 1974), plus the work of Brook^(l973) and Payne 
and cotvor^tefi (1976), indicate that, with the exception' oT preventiw 
services, one ^jhould be cavitious in drawing conclusions about the 
quality of general medical care when studying the management of a 
set 6f tracer problems, diagnoses, drugs, and so on. On the other 
hand, in assessing the quality of medical care for children, one 
. undoubtedly should ^address such areas as the type of preventive care 
provided; 'the content of the case- finding programs; the adequacy of 
diagnostic evaluations; the appropriateness of th^e management, 
including thej>py and follow-up care; and the effect, or outcome, of 
these medicaj care processes on the well-being of the patients. 

In this initial work on develoiing the tracer method, a process was 
specified^by which tracers shlTtihl be selected. In an attempt to 
rationalize the decisioomaking process by which sets of tracersz-are 
selefiCted, we established a hierdffchy of. criteria for use in tracef 
selection. The criteria, in order of importance, were as follows: 
• * ' •* 

— A tracer should -have a significant functional impact; 
— It shoultl be relatively well defined and easy to diagnOs^in both 

field and practice settings; - 
— Prevalence rates shpuld be high enough to permit the collection of 

adequate data from a limited population sample; 
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— The natural history of the condition should vary with utilization 

and effectiv^ess of medical care; 
—The techni(|tiesjof medical management of themndition should be 

well defined for at least one of thevjfbllowing proi^sses: previention, 

diagnosis, treatment, or rehabilitation; . 
—The effect of norjmedical fejftors oi*'the traed? should be undftr 

stood. - ' • . . 

These o^ rial rfiteria have been reassessed; for man,y purposes 
thoy^ are unrealistic and iihpractical because few ip^dical problems 
meet these conditions. Thus the scrutiny of the^ ^valuators is (irAited. 
In another sense, the criteria are rm,^ractical because they ignore or 
rule out niuch of the routine diseise that consumes so much of the 
ph;)^sician's time-. 

In recasting the criteria. w« have tried to focus on medicine as it is 
practiced — the patientS who are seen and their presenting complaints. 
The modi^iied criterta are;: - 

— The^ problem or diagnosis should be of significancejn tercps of 
either its potential functional impact on the patient, the burden on 
. the provider, or its potential ^or treatment resulting-in more harm 
than good; * 
— The prevalence of the condition should be high gnough so that 
analy^s can be carried.out cftn the study population; 
, -^The condition should be^well defined and easy to diagnose; 
—There should be. general* consensus on the medical management of 
the condition for the specific therapeutic process under scrutiny. 

Although we have revised and simplified'' the criteria for selection, 
tfie use of sejts of indicators rather than a single condition is still 
important in attempting to evaluate a health care delivery system. 
.This statement is particularly true gfven the lack of concordance in 
^.th^ outcome among a set of tracers. However, in the past 3 years, 
four articles have come to my attention in which the **tr^r 
technique," of a similar approach, was used to evaluate health services 
(McBride^and Ralph 1976. CampbeJI 1974, Smith 1974, Novick et al. . 
1976). In all these published reports, a single problem or diagnt^sis 
..was "the tracer" used. Several articles indicate that a set of tracers 
had been selected, but data were reported on only a single condition. 
It is possible that inadequate funds or personnel forced tht authors 
to limit their study to one conditio^ 

LIMITATIONS OF Trip FIELD^STUDY 

The preliminary prospectus for this study was .devdoped between 
1969 and 1970. It v^as clear then, as it is now, that tj^ study results 
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\ • # ■ * . • 
ik^could not defirtilively answerptl^e question **Is the healtK^statu^ of a 

child relatjed to rtie organization that provides the child's usuaU 

medical care services?" Rathep^ this study was desigurt- primarilyrTto 

mea^jrt^ the. health. Jtatus. a selected population; asse^ differences 

in provider performance, qrganization, and attitude/ and provide 

"^'•information about ^the strengths and weaknesses of ^he tracer viri^thod. 

Becaus^ff^^ese data may be used inappropriately to^jnake narrow 

cq/nparisons of particular populations of children or providers oY to^ 

draw ve^y broad concUj^^JAs ^bout the delivery (if health service^, the 

limitations of thc^tudy must ^learl)? stated, ^ \ 

The most crucial limitations are its .location^ in one city; ^its^v 

predon»inantly*black sample; its repre^entatioij.of f'afnilies fromthe^ 

community and not patie^nts of a defined group of'jShysl^ians; and. its 

failure to include all types, of providers in the record feview. With the 

exception of the sampling procediu^qs employed, these limitations^ 

arose, ftom cost and.logfs;ic consiaenitioris; These factors should* 

point out the necfl f6r c^ion jVi applying these r^ults to other cities 

an4 other populations. x* ^ - J. 

• ^ ». ■ . 

It IS important to note that while we divide providers into groups 
such as solo practitioners* ^sst)cfi^nons of p*hysicians, ^d public 
clinics, we recognize, that within eacBTof these categories th^re is 
heterogeneity. Our analyses, therefore, focus on categories of provi- • 

(ders anti jot on individual orgapizatioryj or practitioners within a 
single category. Further, we cannot know that af particular child used 
one ^ovider exclusively. We separat^ed the data on those' children^ 
. whom we could identify as using two different l^inds of provider 
organization; but within a provider category, potably the- hospital 
outpatfent departments, some of the children very likely may have 
used more than one facility within that provider group. 

In addition to these factors related to the broad design of the 
^ study, there are limitations-^ the data that were collecteQ. Scrupulous * 
attention was given to calibrating and checking instruments during 
the clinical examinations to avoid erroneous findings; a special effort 
was made to .gather follow-up data on a random sample of children 
who parsed the screening tests so that false- positive and false-negative 
rates could be calculated;'and an effort was made to standardize the 
findings of the- physician observers. In several initial meetings, the 
terms to be used for anatomic descriptions were discussed, and 
several children were examined by all members of the ear, no.se, and 
throat staff. Beyond this initial training session, we were unable, 
during the course of the clinical examiaations, to effectively mHSisure 
interobserver or intraobserVer error. 

It is necessary to clarify^ differences in the way prevalence rates 
were compiled between the clinical field study anc^ the chart review 
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dTiaiyfti. During the tlinical ^aminatiofi,, we determinfed point 
prevalence rates: thai is, thejj4imber*of children with the condition ^t 
. the'time they were examined, relative to the total number of children 
^^mncd. Duri^Hj the medical record review, howeter, the data.we^ 
coTlectecT defined the number of children with any evidence in thdr 
records^f the condition at any tniie'in the past: We anticipated that 
-these latter r*«s, which _we termed variable-period ^evrfence rates, 
would-be higher thaq'tMe pomt prevalence raies because the children 
could have had one or more episodes of the-viisease during the 
longer period of tinie and i^cau^e ^he providers had several oppor- 
tunities to-detect diseases, ^ ► . 

In the strict sense, point preivalence rates and Variable-period . 
^ prevalence ratea cannot be compared^They are useful, however, as 
vindicators of tht case-finding activities of the health care providers. 
If, for example, the variable-period preyalenpe. fates' are lower^ when 
logically they should be much higher than the.point prevalenc^ates, 
one can be confident that many cases have not been detected by the 
providers or have not |3e,e'n entered in the parents' charts. 

Complex problems arise from ascribing certain untreated condi- 
tions to a child's current provider, with both highly ipobile popilla- 
tions and new provider organizations; however, this is also true with 
relatively stable patient ^nd provider groups. For example; if a child 
has a conductive hearing loss and scarring of the tympanic mem- 
brane, indicating that the loss is associated with previous middle ear 
infection-^^p>»dr&ps an infection that was pogrly managed 4 to 6 years 
earlier— it may be inappropriate to assign poor manaflkment of the 
early infection to the present provider. Further, it must be recognized 
that, in this study, we measured the health status of the children at 
only one point in time. Thus, we cannot ascribe directly to, any^ 
provider a c'h^nge in health ^status of children to whom he delivered/ 
care i ■ ) ■ " ' ■ [ 

Finally; knowledge of the nature and severity of a <Aild's healtA 
problem could have in>luced parent*^ to take their'child to one^ 
particular type of provider rather than to another. Such self-selection 
factors could result in* the presentation to di'fferent provider groups 
of biased samples of children— an unusually large or small number 
wit^ serious problems. This indeed would' be a concern if the results 
of this study depionstrated that, among the types of provider 
organizations, there' were significant ditferences in the health status 
of children ol>^bmparaSle social and demographic characteristics. 

These are soije of the specific limitati^s of this large community- 
based evaluation program carried out in the early 1970s. But what is 
the status of quality assessment^in 1977 in North America? Current 
interest ui-^valUation and pressures to develop quality assurance 
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programs have increased exponentially during the past-5 yfears in the 
face of rnouhting data indicating that feasible, valid, and reliable 
methods to^assess health services have yet to be developed. Clearly, 
we are far from being^able to integrate an effective and efficient 
quality assurance mechanism into a health^care system. 

To begin vtith, we constantly face the nagging question — What is 
quality? In a -recent paper. David Rutstein and his colleagues ( 1 976) 
defined quality^as being concerned with the .outcome of cafe. 
However, in common parlance, "quality** means degrte of excellence 
and degree of conformation to standards. Xlie issues in medicine are 
•excellence in whose terms^and conformation to ufhose standards. 
^Quality is not a static concept; it is dynamic and it relates to the 
reasons for the assessment, the methods employed, what will be 
measured, ahd how it will be measured. 

At a more pragmatic level, we must cope with the unreliability of 
recorded diagnoses. Criteria must be applied to specify who has a 
. streptococcal* sore throat or a behavioral disorder. Having applied 
these criteria, we are obligated to evaluate a sample of those who do 
not meet the criteria as well as those who do. for in the former group 
a high proportion of inappropriate care may be* foufljii. 

One of the most difficult issues in assessing child health care from 
xisting medical records is defining an ^isode of illness. Is it 
ossible, retrospectively, to put a time frame around an episode of 
emia. otitis media, or a urinary tra^t infection? Of course these 
lineations can be made with extensive protocols and a prospective 
esign. However, we must be concerned with- the cost of evaluation, 
and thus we must ask what can be accomplished with existing data 
and a p^-agmatic methodology. 

Current inpatient audits in the United**States are estimated to cost 
$250 miHion annually (New England Journal of Medicine 1977). The 
cost of a simple evaluation based on encounter form data was 
approximately $2.50 per encounter (Kessner et al. 1975). Even if 
such an approach were Applied to a sjnall sample of all outpatient 
encounters, the cost for the I billion ambulatory care visits made 
annually in this q)uhtry would exceed $250 million, and this figure 
does not include developmental costs. I should add that such a **bare- 
bones'* process evaluation would not include many of the broader 
approaches now advocated for evaluation studies— total episode of 
illness and inclusion of lifestyle, social, psychologic, and environmen- 
tal factors. These approaches are pipe dreams. Even with funds and 
physician and patient compliance, we still would not have reliable 
tools to measure many important social, psychologic, and environ- 
mental factors. ^ . 

Given that we could develop and implement a valid, reliable. 
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simple, and^irifecipensive guali^y assurance^ system; and given that the 

data will 'tc analyzed in '^'{ii^iely 'fashion and ih a format that js 

^eanirfgful to .phy;^ician3 ai'ffd administrators; and given that health 

professionals will change Jheir organization or clinical practices basied 

on these data — will thtre be a measurable impact on the health status 

of our patients? There is very little^evicjence to suggest that there will 

(McNerney 1976). 

Clearly, this is not the time to expand existing quality assurance 

systems or to initiate complex and new programs. Ratl^r, wfr must 

now carefully assess ^the impact of our present efforts on both the 

cost of health services and the -well-being of our patients. Child health 

may be served H>etter now by expanding carefully* studied and 

efficacious pediatric preventive and*case-finding activities than by 

prematurely joining the rush to establish complex quality assurance 

systems. * - 
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DISCUSSION 
Maurice Backett and Barbara Starfield - 

Dr. Kessner*s paper described a quality assessment study with a 
very dear objective;, to examine the adequacy of different types of 
medical care services. Thus, the study clearly fits into the category of 
effectiveness studies, which e;camine the relationships among struc- 
tural, process, and outcome variables. TTie study was population- 
based rather than clinical in that the sample was drawn from the 
community. , , 

Thi^^ajor limitation of the method was its very high cost, 
exceeding $1 million for evaluation of care for only a few clinical 
"tracer** conditions. * ♦ 

Implementation of the findings was problematic. Although the 
study clearly invqlvefi^quality asses^ent rather than quality assur- 
ance, the investigators (and especially the sponsoring and funding 
agende^) appeared to have an underlying, much larger expectation 
thay^efects found in the assessment would be acknowledged and that 
these would lead to changes,, either within individual practices or 
through actions at higher levels (such as professional associations). 
Although this expectation was never stated explicitly, Dr. Ressner 
believes it existed in this stu^y as it does in all similar studies. If such 
expeci^tipns are justifiable (there were many who felt they are not) 
and if ^ researchers are to avoid extreme frustraticrK, much greater 
effort to involve practitioners or practitioner organizations earlier in 
the planning of the study concerning their practices is required. 

In the discussion of this study, there was general agreement th^t 
the method had applicability in different areas and cross-culturally, 
although the specific tracers would have to be different. However, 
two cautionary notes were sounded: (1)^ Efficacy of medical' care 
interventions for the particular conditions must be documented 
before a study is mounted, and (2) The tracer concept must not be 
misused. It is not justifiable to draw conclusions about the overall 
quality of care by examining the care for just one condition, as there 
is as yet no evidence that the presence of one defect (or one type of 
defect) in a pr^tice is associated with the presence of others. 

As a result of his study. Dr. Ressner .has concluded that current 
quality-assurance activities lack justification because of the unavaila- 
bility of a valid, reliable, sensitive, an^ inexpensive measure of quality 
which can be applied on a broad scale. Instead, he would give priority 
to efforts at prevention and case finding. Vjewing his suggestion in 
another light, perhaps it might be concluded that quality-assurance 
activities are misguided because they fail to address the most 
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important areas — these include the inadequate recognition of prob- 
lems in the community and failure to assess the outcomes of care in 
individual patients. «l 

All quality-assurance approaches^ focus on documenting the validity 
of diagnoses and the prescription of appropriate therapy (whether 
judged by validated or unvalidated criteria, or by explicit or implicit 
judgments) for diagnoses. Quality-assuranCe organizations rarely even 
consider the possibility that practitioners have misdiagnosed, as most 
choose patients with certairf diagnoses as the basis for review of 
quality. Moreover, reviewing the extent to which desired health goals 
are attained as a result of medical care intervention is given little 
attention. Perhaps arrangements by which success and failure in 
achieving expected goals in patient management on a day-to-day b^sis 
can' be determined would do more to improve the quality of care 
than sporadic, retrospective studies which demonstrate current de- 
fects. 
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A SIMULATION MODElt- ^^R THE EVALUATION OF 
THE MATERNAL-CHILD CARE PROGRAMS IN 
' ^ COLOMBIA 

Rodolfo A. Heredia, Alejandro Vivas, Luis Hernando Ochoa 

INTRODUCTION ' * 

■ ' \ 

. This paper describes a preliminary simulation model for integral 
evaluation of Maternal-ChilcJ^jGare Programs (MCCPs) in Colombia. 
The model includes both maternal apd child aspects, reflectilTg the 
priority given them in public health and development programs by 
the Government. 

Integral evaluation is achieved by incorporating the Maternal-Child 
Care, Model (MCCM) into a more general model, known as SERES* 
(System for Study of Economic, Social, and Demographic Relations — 
Sistema para el £studio de las /ietaciones fconomicals, Sociales y 
Demograficas). This approach, in addition to controlling, aspects that 
belong exclusively to the MCCPs, allows for the incorporation . of 
other health-related elements, such as econonfic factors, population 
structures, and gpvernment acdyities. ' ^ 

The idea of using a simulation Xechnique to develop an evaluation 
instrument for the MCCP arose during the presentation of SERES at 

. the Maternal-Child Care Division of khe Ministry of Health. At th^t 
time it was agreed to test a new methodology in the field, using an 
interdisciplinary team of pediatricians, gynecologists, public health. 

, specialists, sociologists, and economists from tMOth a private institution 
(CCRP) and the Ministry of Health. The group endeavor was to 
formalize the MCCP so that it wpuld, eventually materialize into an 
MCCM proposal. The model was to incluci^e the most acceptable 
methods tind techniques, given the restrictions of available data and 
data colleuion procedures. 



' SERES is a simulation model developed at the Corporacion Ccntro Regional de 
Pbblacion (tCRP) (Regional Population Center Corporation) for analyzing '(through 
comparative studies) the impli^ation|f for 'development of different poHo" alternatives. 
Appendix 1 contains a brief description of this model. , . - 
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^ The teani began its task by ahalyzmg the SERES health submodel 
and by studying in detail the different components of the Govern- 
ment MCCR Unfortunately, the orfgo'mg study was interrupted by a 
medical strike that affected the country during the last quarter of 
1976, Up to that time, k good deal ^f work had been carried out, ar^d 
the discussions dealing with a theoretical 'MCCM (similar to the 
SERES health submodel) had started. tVhen the strike occurred, the ' 
CCRP group continued to work by itself, so that the MCCM a/id'the 
information presented here' essentially correspond to the CCRP 
effort. It is^' hoped that meetings with the MCCP team will start again 
in the near future. In this. way. the great advantages perceived and 
the results obtainefd throiigh the interaction between the "technicians 
.and adminfctrators" can ^ used. 

THE EVALUATION PROCESS 

The need, for evaluation of Government programs in Colombia 
does not -seem to be debatable. On the contrary, technicians ancf 
administrators favor such programs as^much as they strive to Resign 
better instruments for the purpose of evaluation. This concern comes ' 
^ from an awareness that — in countries with large and rapidly growing 
populations, great accumulated* deficits, and inadequate social and 
^ecpnomic structures — it is necessary to set priorities among programs 
and. to rationalize by using the limited resources available irt order to ; 
make substantial short-term improvements. Interest in the evaluation " 
aspects has increased with the priority recently given to the health 
field (DNP 1969, 1970. 1975; Ministerio de Salud Publica 
1967a. 1973). but as it will be shown, although the country' l\as 
• adopted the evaluation component in the programs, it has not set a 
clear definition of evaluation, nor has it included^appropriate instru- 
menyts for its implementation. 

Difficulties concerning evaluation do not belong exclusively to 
underdeveloped countrie.s. Turner (1976. p. 26) states- in general:* 
"This rapid growth (of program evaluation) inevitably has beeh 
accompanied by confusion about purposes, methods, organizational, 
arrangements and. riot least, terminology." Roemer (1971, p. &39) 
states: "Because of the complexities of evaluating methods of health 
service organization, there has been a great deal of confusion^ in even 
deciding what should be evaluated, let along how to go about doing', 
it." ' . ^ . 

These observations have led to a partial review, of Available 
"^literature. This review was intended to provide a reference frame*-.* 
work that would permit the analysis of the. content of evaluations 
carried out or planned in the MCCP field in Colombia and would 
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determine the possible role of the proposed model in the evaluation 
process (Donabe^iaj;! 1966, Freeborn and Greenlick 1971, FPEI 1974, 
Roemer 1971, andf Turner 1976). 

The review indicated that it is possible to design a framework based 
on three general evaluation levels corresponding to the different 
program development stages. The first*^ level assesses the need for a 
program with respect to other programs such as health versus 
education. A second evaluation level refers to the available design 
alternatives for the .same program — for instance, the different com- 
bination of instruments for health care delivery. The third level deals 
with the program's implementatipn (figure 1). The first two levels are 
basically analytical, and from them the standards and expected values 
of the program are produced. Their particular features fuse them 
with the^planning process. The last level reflects the actual develop- 
ment of the program being implemented. The evaluation is achieved 
through comparison of the different levels and not, as is usually the 
case, through a description of results or their relationships. , 

Using the definition given by Reynolds (FPEI 1974, p. 4): **Evalua- 
tion is a process for making judgments about selected objects .and 
events by comparing them with specified value standards for the 
purpose of deciding ainong alternative courses of action." 

The comparision between the different levels can be expressed in 
terms of evaluation subjects, each corresponding to one of the 
analytical units into which ayprogram can be divided.^ Those most 
frequently mentioned in the literature are inputs,^ processes, results, 
effects, and impacts. Inputs refer to resources used to render servic^ls — 
such as administrative F)ersonnel, physicians, nurses, equipment, and 
so on. Processes rtfer to the "basic structure- and regulations of the 
service system — administrative regulations, document flow, and orga- 
nizational ranks are some examples. Results deal with the type and 
.frequency of services renderctl, such as X-rays, medical examinations, 
r^and so on.rEffects and imp^t refer to the repercussions of results on 
I other aspects within or out.side the health system — for instarfce, the 
. patient morbidity rate or the population age structure. 

The evaluation is carried out through indices derived for each level 
at each evaluation subject category. It may be said that there are two 
Types of index: those of effectiveness and those of efficiency. The former 
link up' with the outcomes of a given evaluation subject; the lattej, 
with the relationships between outcomes of different evaluation of 
subjects. Indices pf the first type dre measures such quality of 
service, percentage of population well* treated, number of medical 



^ ' There are no uniform definitions ior the level? and evaluation subjects used in 
evaluation literature. Those used here are based on Reynolds* definition. 
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Figure 1. Levels and subjects of evaluation [From CCRP, Area Socioeconomia, Modelo Materno-lnfantil, March 1977]' 
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examinations by different category of services, and so forth.. Aa for 
the second one, we could mention the relationship between services 
rendek'ed and inputs used and impact and expenditures.' These 
indicators are usually part of (;Qst-benerit studies. 

HISTORY OF EVALUATION IN COLOMBIA. 

In order to understand to y^hat extent the model satisfies MCCP 
evaluation needs in Colombia, it seems appropriate, within the given 
reference framework, to present a brief critical rcjview of the types of 
evaluation and indexes used in evaluations that have been proposed 
or planned so f^r in Colombia. ' - 

During the past years, the need to evaluate pro-ams has been 
recognized in the health system. The first . step toward the introduc- 
tion of evaluation techniques was taken when health system planning 
was institutionalized in 1968. This event took place as a response to 
the commitments the country had made to the First l/ecennial Plan 
for the Americas (Ministe^o de Salud Publica 1967a). 

The Colombiaii health plan for the 1968-77 period was based on 
th^ Survey, of Human Resources for Health and Medical Education 
(Ministerio de Salud Publica 1967b) and on the National Inquiry on 
Morbidity (Ministerio de Salud Publica 1966). Thfe plan showed the 
magnitude of morbidity risks and mortaKty levels of mother and child 
populations. In response to this situation, the\nother-<hild group was 
created as part of the Ministry 6f Health in 1969. Later, as part of 
the development of the 1972 health plan, mother-child care was 
given first priority. In ^1974, family planning was added. As a result, 
the Maternal<|Ahild Oare and\Population Dynamics Division Was 
, established in Ministry of PuBlic Health. ^ 

Although pumning of a Colombian health system began in 1968, it 
was not^ until 1975 that the first formal evaluation of the MCCP 
activities was attempted (Ministerio de Salud Publica 1975a). The 
second evaluation was carried out in the middle of 1976 (Ministerio 
"de Salud Publica 1976a). Additional evaluations cover more specific 
aspects of the MCCP: for example, the followup studi^^f family 
planning acceptors undertaken since 1974. ^"N^y ' 

Up to* the present time, general evaluations carried out in Colombia 
by the MCCP have concenttated on one evaluation level, that of 
implementation. The.treatm«ht of M(pCFs impact aspects on health 
and other socioeconomic Elements was made on a speculative basis 
only. From the standpoint of comparison standards or norms, the 
1975 evaluation made explicit only those aspects relating to geo-* 
graphic^nd population coverage. In 1976 the standards for coverage 
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estimates by auxiliary nurse and fleldworker activities were also 
clearly stated. » 

Because of design changes in the Colombian National Health 
System (NHSjl begun in 1973, the importance of a permanent 
ev^uation process for the- ministry's programs (Ministerio de Salud 
Publica 1976b), as well as a systematic compilation of data required to 
ensure feasibility, became apparent. The plan proposed by the NHS 
seeks to evaluate the outcome of programs for some of the evaluation 
subjects previously mentioned. 

The resulting system, completed in 1976 but not yet implemented, 
p;roposes the generation of both qualitative and quantitative indica- 
tors. The qualitative indicators have been assigned different labels: 
precision, satisfaction, emphasis, extent, ^fTicacy, and effectiveness of 
health services. The quantitative indicators are the following: physical 
and monetary resources, program activities, persons who received 
care, population and service coverage, and activities per person 
(concentration). Efficiency indicators proposed by the NHS are as 
follows: cost indicators (monetary resources by activities) and yield 
indicators (activities by resources). 

The evaluation system proposed by the NHS represents important 
progress with respect to the 1975 and 1976 evaluations, because it 
explicitly states the' indf|^tors for each selected evaluation subject. 
However, there are still imperfections when the system is compared 
with the previously mentioned reference framework, 

T~The evaluation is limited to program implementation and does not 
deal with need or design aspects, 

— The comparison standards or norms are mentioned only for 
coverage and concentration ii\jdicators, 

—Indicators selected for evaluation fundamentally emphasize the 
effectiveness of progfr.am achievements. Therefore, evaluation of 
effectiveness and/or efficiency of other aspects, such as the pro- 
gram's inputs, effects, and impacts, is neglected, 

— Finally, the use of such terms •as "effectiveness," "efficiency," and 
"effkacy" as indicator classifications in the evaluation system leads 
to confusion and differs from the use generally accepted in the 
field of Evaluation, 



THE MATERNAL-CHILD CARE MODEL 

Nfodel-building is not foreign to the field of health, where mathe- 
matical simulation models have been developed. Generally, however, 
such models have been directed toward specific aspects of the system. 
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such as hospitals, outpatient visilfs, and so t)n (Edwards 1971, Lindau 
1972, Feldsteirj lj967). They have rarely been aimed at the study of * 
health within the more general structure of the health system of a 
country.* 

The preliminary version of the MCCM is based on the structure of 
a more comprehensive health model (CCRP 1974a»b), sqme aspects 
of which have already been integra|pd into SERES (CCRP 
1975,19745^ Within the context of the integral evaluation of the 
MCCP, the MCCM is incorporated in SERES as 2i programatk unit 
(figure 2). , % , 

The model for* maternal-child care has been developed in two 
general stages. In the first, the different classes of demand^ and 
service requisites are calculated. In 'the second, the standards or 
tiorms of inputs,, results, and effects are determined. 

First Stage — Calculation of R»al Demand and Services 
'Requirements 

Estimation of Demand Levels 

Selection of, risk groups within the target population. The demographic 
model (on a dynamic yearly basis) generates data on population, 
composition by simple age groups, births, and sufficient data to 
estimate the numbers of married and pregnant women. The target 
population is selected within the model, bearing in mind government 
priorities. It is then formed into risk groups by age, sex, and urban- 
rural residence; • ; 

Establishment of potential demand {number of cases in the country, by 
morbidity rates). Based on risk groups, cases (individuals) are estimated 
by type of disease, using the appropriate morbidity rates which could 
be seen as a result of an epideipiologic transition process and of 
preventive programs. In estimating demand, two points are worth 
emphasizing with i;espect to the determination of morbidity rates. 
The first is related to the epidemiologic transition (Omran 1971), and " 
the second refers to preventive programs. 



* N(odcl»building in Colombia should take into account research by institutions 
focusing on aspects of the health system: tl^e National Health Institute of the Ministry 
of Health/ the Colombian Family Welfare Institute, the Department of National 
Planning, the Antioquia Health Planning Program ^f the School of Public Health, the 
PRIMOPS ( Pro^rama de Investigacibn en Modelos de Prestacion de Servidos de Saliid) 
program of the Department of Valle, and COLIMPLAS (Programa de Investigacibn 
en Planes de Salud OMS-OPS). 

* The term •*demand** is used instead of "need" to avoid cross reference with one of 
the evaluation levels. ^ 
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Thc^pidemiologic transition permits understanding t^e observed 
relationthip between incidence and prevalence for certain diseases 
and the level of development of a country (Puffer and Serrano 1973, 
p. 309; United Nations 1973, p. M8). The ability to understand this 
phenomenon is of great importance in determining the demand for 
scrvicesr— especially if one considers the precarious life conditions of 
the population,, particularly as they- affect children. The need to 
integrate the model (with indicators not included>in the MCCP or in 
the health system) is derived , from the nature of the described 
phenomenon: The health status of the population not only depends, 
on direct medical attclntion but on other factors, such as levels of 
education and nutrition, housing conditions, the existence of potable 
water and waste' disposal systems, and so on. Under certain conditions 
these infrastructure elements can have a greater > effect on< the 
morbidity-mortality of a gyoup than do health programs. 

Preventive programs may eventually accelerate changes in the 
morbidity structure so that a massive vaccination campaign can affect 
the number of infectious disease cases independently of the level of 
development of the country. 

i*€U demand (number oj cases that perceive the need for health services) and 
real demand. From potential demand estimates, it is now possible to go 
into felt and real demand, using fixed coefficients which include the 
patient's perception of his/her health status and the geographic 
location of the services'. Note that the term "real demand" is being 
applied to individuals who are conscious of their status, have 
favorable attitud^es toward public or private health services, and have 
geographic access to them. (The cases which will become the respon- 
sibility of the MCCP are estimated in the next step.) 



Estimation of Requirements - 

Calculation of physical requirements b^^saHsfy-'-feal^'uemand. Based on 
information concerning real dem;2md, it is possible to convert this 
demand into inputs by applying a matrix of inputs and activities 
needed to treat each group of illnesses by cause. This combination of 
resources is defined by the existing or available technology for the 
country at a given njpment. Specification of the technological proc- 
esses is of particular importance to planners. It is here that the 
activities and inputs required to treat a patient (case) are defined. 
Alternate resource combinations, such as increased participation of 
paramedical personnel and systems specialization, fall into this cate- 
gory. (Each alternative technology which is studied generates differ- 
ent pressures on the health system and its resources.) ^ 



^ 
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Adjustmentr of physical requirements. The following step determines 
whether the society has available inputs or can secUre them during 
the process. In order to accomplish this, tfi^ limitations of the country 
in terms of human resources, equippient, hospitals, and so on shj;2uld 
be made explicit. This estimate leads to the adjustment of physical 
resources needed to meet real demand. 

The availability of human, physical^ and other resources is/^f 
special importance for the achievement of MCCP objectives. Many of 
these resources are not under the control of the health system, but 
depend on the development of other sectors in the country/The 
integration of the model with* SERES permits the evaluator to verify 
explicitly whether or not society has the required program inputs. 

Calculation of requirements in^monetary units. Next, the physical 
i;equirements that were calculated previously are expressed in mone- 
tary units, using information on average cost for inputs. In this way, 
the model generates information on the total cost Incurred to provide 
total maternal-child care services. Part of this total cost Will be covered 
by families and employers, and the Government will finance the rest. 
This situation implies calculating non-Government expenditures and 
subtracting 7hem from the total cost in order to obtair> the cost for 
the MCCP— that is, Estimating the purchasing power of the popula- 
tion and its access to certain health subsystems (social security, health 
insurance, prepaid medical care, private services, and so on). Given 
the cafJacity of SERES to generate information on income distribu- 
iion, consumption patterns, and employment, it is possible to incor- 
porate the effefts of changes in these variables .on the demand for 
public services, 'Nevertheless, ^he difficulties implicit in estimating 
demand functions under public expenditures ar^ recognized, 

JiECOND Stage— EsiiMATiON of Eftective Demand, Mortality, and 
D^YS OF Incapacity ( I able 5) 

The second genei^^l deyelop^ment stage of thi mqflel consists of 
determining those cases which can be effectively tr^ate'd by the MCCP 
(effective demand) and calculating the effects of \he program on 
maternal-child mortality aitd disability. The main step^ are: 

Determination of Effective Demand and Achievements 

Effective demand is the result of the comparison of monetary 
requirements (to meet the real demand) with resources available to 
the MCCP (which come from the Government). If the resources are 
greater than the requirements (in monetary terms), the level of 
effective demand will be determined by real demand. If the reverse 
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is the case, then effective demand, will depend on the limitations 
imposed by. the volume of resources, as well as on the priorities 
among MGCP subprograms which have been se"t by Government 
policy. Effective demand, ^in terms of casei^and r^ources, must also 
be interpreted. In this instance, the previously calculated require- 
-ments are adjusted according to the policies and economic capacity of 
the Gf>vernment. -il^ 

Case classification and^ype of Care (attended, poorly attended, and 
not attended) 

In the second step, cases are classified into two groups, those who 
were not treated and those who received popiV can^. The second 
group IS made up of three types: . 

— Cases resulting from the difference between potential demand and 
actual demand (interpreted as cases without access to the MCCP 
system); - 

— Cases obtained by subtracting effective demand from real demand 

. (interpreted as cases that lack MCCP coverage); 

— Cases poorly attended (calculated by applying a coefficient of rhe 
effectiveness of attention to the cases attended by the MCCP). 

Estimation of Mortality and Disability 

J • 

Once the two case groups are defined, appropriate mortality rates 
are applied to gen^rate/child-matei'nal mortality levels: A coefficient 
of illness duration^by rrek groups is also applied to the two groups. In 
this way, the total number of days absent from work or from school 
can be obtained. The most sensitive^ aspect of this stage is the 
calculation of the morbidity and mortality rates and the duration of 
illness, especially for the group of ^tases not treated. 

The natural rates are known for only a few diseases, and these 
have no known^ffective therapy. In order to obtain a natural rate, 
an experimental population (subject to disease but not receiving 
medical care) would be required. An assessment of cause of death 
and duration of illness would also be necessary. These conditions are 
not possible in real-life situations. 

In order to measure morbidity arid mortality in well-treated cases, 
rales corrected by the effects of treatment should be applied. These 
coefficients will be obtained from institutions with "captive popula- 
tions" which are receiving adequate health carie* It is understood that 
such coefficients are numerically less than those for the rest of the 
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cases; therefore, disability and death estimates would also be lowei 
among these groups. 

4 

Indicators of the Matbrnal-Child Carb Model 

- ^ ' ■ C 

Through its different stages, the MCCP can produce a set of results 
that can be converted into evaluation indjjcators. These indicators are 
subject to limitations of the MCCM and SERES structures, and they 
are pi^oduced by the ^^sen design alternative,* This alternative, in 
turn, is defined in accordance with Government goals and technolog- 
ical considerations. . * 

Measures derived in this way can be seen as "expected values" of 
the different elements relevant in'the study of the MCCP, Within the 
proposed evaluation schj^me, these, measures correspond to standards 
produced at the needs and design levels and should be compared 
with the indicators produced during the implementauon leveK The 
indicators may be grouped according to the different steps iotor which 
the model has been divided, as follows: 

/_ , 

Indices at the Demand Step 

Most indices estimated at the demand step (figure 3) belong to the 
needs level (figure 1) and can be interpreted as effect and impact 
goals. 

Demographic indices. At a risk-group level, these indicators permit 
the study of size and changes in age groups^of the MCCP target 
populations at the national level. Changes in such indicators afe 
related to changes in the demographic behavior of the population 
(fertility, mortality, and migration), some of which are caused by the 
MCCP (maternal-child mortality,- family planning) and others, of 



* SERES was specifically oriented toward the study of design alternatives through 
the socalled control elements (shown in figures. 3, 4, and 5 by means of small, directed 
rectangles). In general, control elements can be divided into two groups: policy 
controls, which are specified in order to study the alternatives in Government 
decisionmaking, and parameter controls, which must be specified to Calibrate the 
dimensions of the model. The latter are necessary to the extent that constraints exist 
on our knowledge of th^ behavior within the MCCM or of its interrelationship with 
the rest of the system. 

When calculating demand, the first type of control embodies the parameter named 
''Government definitions on maternal and child policies foi; the country." The second 
type deals with morbidity rates, epidemiological transition^ preventive services, and the 
attitude and accessibility coefficients. When assessing requirements, policy controls are 
those parameters which define the input-activities matrix. Availability of resources, 
average cost of inputs, and non-Government expenditures are elements of parameter 
control that adjust to the model. 
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which occur outside the MGCP (changes in education level, employ- 
ment, life expectancy, and so on). 

Health status indices. At the potential or biolpgical demand estimation 
step, these indicators^show data on clinical morbidity or clinical needs. 
, They refer to the incidence of different diseases sustained by the 
target population, regardless of desire' for or availability of health 
services. / 

Cultt^ral indices. At the fcflt demand estimation step, these indicators 
show information on the felt morbklity and needs, that is, xhe 
different diseases experienced by the individuals as perceived by the 
population once their criteria are considered. 

Coverage indices. At the real demand level, these indicators show the 
number of cases to be attended by the national MGCP; the number 
of cases is calculated by taking into account structural aqd cultural 
restrictions as welf^as Government health i!>olicies. 

Indices at the Requirements Calculation Step ' 

Useful indicatoi-s for evaluation generated at this step of the model 
(figure 4) are complementary to those obtained during the estimate 
of different der^ands (need evaluation level [figure 1]), They cap be 
grouped according to the measurement units employed (either 
physical or monetary). 

* 

—Physical indices (obtained during the calculation of the total and 
adjusted requisites). ^ 

Inputs and activities indices. These indicate the resources necessary 
to satisfy real, demand in terms of the quantity of inputs (profes- 
siorv^l and paraprofessional personnel and auxiliaries, equipment 
and buildings, drugs, diagnoses, and administration) and activities 
Specialized, general, and simplified consultations; and specialized 
general hospitalization). 
Q ^/ indices, these identify the pressures on necessary 

resources to satisfy real demand and those inputs which need to be 
promoted in the country. 
— Monetary indices. 

Indices of cost of inputs and activities. These give an idea of the 
maximum cost that could be incurred by the maternal and child . 
demands in the country, given the resource constraints and the 
level of technology. 

Indices of cost for the MCOP, These are similar to the maximum 
cost indicator but are applied to the MGGP. (It is assumed that the 
Government is responsible for absorbing the value of the residual 
services; once the non-Government expenditures are deducted). 
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Indices at the Steps of Estimation of Effective Demand^ Mortality^ and 
Days of Incapacity ^ 

At these steps of th^ model (figure 5) information on expected 
results (inpi^ts, activities, and effects) is generated at the design level 
(figure These indicators are very closely related to the different 
measurements normally used to describe the operation of programs. 

Deficit and surplus indices^ These are derived through the comparison 
of the monetary requirements of MCCP and Government expendi- 
tures on the program. They point out the need for additional 
Government efforts to cover the full cost of the MCCP programs and 
to modify its expenditure priorities for MCCP subprograms. They 
also indicate the possibility of improving the quality of services where 
surplus of economic resources exists. 

Treated cases indices. These indicators can be seen as "positive" 
health measurements. They result from a process that first calculates 
demands^ then estimates different resources available and finally 
compares both elements. These indicators are specified. according to 
the different risk groups considered in the model or to the type of 
care^provided (well and poorly treated cases). 

Requirement indices. These are some of the most commonly used 
standards for evaluation. They refer to the quantity of inputs (human 
resources, equipment, and fixed capital) required to satisfy the 
treated cases. 

Coverage indices. It is possible to obtain these indicators both for the 
volume of inputs and activities and for treated cases. They are 
derived from data obtained during the preceding step and from the 
calc^ation of demands. Because the MCCM is^ incorporated into 
an integral interpretation of the indices is possible, since, they 
take into account the demographic, economic^ and social conditions 
^- of the country. ' 

Mortality indices. Changes in the mortality r^les hav^ been used as 
the most frequent "negative*' index of progranj^ resutls. 

Iv As the MCCM has been specified, the 'mortality indices take into 
^^ccount differential mortality rates according to Tiealth services 
received by the population, defined in;*^terms of coverage and 
classification of cases. These indices can be separated into perinatal, 
infant, and maternal mortality rates. 

Disability indices. The cost*of illness to society is observed through 
days of absence from work or school/The model generates informa- 
. tion for this index by applying coefficients of duration of illness to 
the "target*^ groups. y ■ 
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Final Considerations 

Fof evaluation, the model' structpre has been specified so as to 
include the indicators derived from jhe review of the 1975-76 MGCP 
evaluations and from the proposal incorporated into the Colombian 
health system.- As it was shown -above, the model responds to the 
different evaluation levels and objectives proposed in the framework. 

Incorporation of the MCCM into a more general structure. such as 
the SERES model allows for dynamic feedbacks of MGCP with 
demographic, economic, and social determinants, thus permitting 
derivation of additional indices at the impact level. Since it is usually 
not possible to integrate the MCCM into a general model, the 
elements thavdefine the relationship between the MCCM and the 
rest of the system -must be isolated and treated as exogenous 
conditions; this task can b^» accomplished by using information 
generated by other models or by experience. ' ■ 

Formulation of the MCCM in term of three steps — demand 
estimation, calculation of requirements, and adjustment between 
demand and available resources— provides a general framework that 
can be applied for formalization of other topics at different levels of 
complexity. 

In. addition to program evaluation, a model like MCCM can be 
"used by planners to compare project design alternatives. For this 
purpose it is necessary to specify the values of the political, technolog-. 
ical, and servitie- parameters that define each of the alternatives. Each 
alternative will generate a different set of values for the.indices of the 
model, and comparisons among design alternatives are achieved by 
contrasting these sets. . 

The possibility of developing the model as. an educational tool 
becomes an important issue during the work of the interdisciplinary 
team. Participation \)f the members increases their knowledge and 
interest in MCCPs. It seems that the specification of a clearly 
attainable goal; the systematic pattern of thought imposed by formal- 
ization; the joint review of the program; the need to identify the 
relationships between MCCP elements (both internal and external); 
and the incorporation of demographic, economic, and- monetary 
resources a.s well as other limitations-~all lead to a. better understand- 
ing of what constitutes the MCCP system, illustrate the importance of 
trade-off considerations among policy alternatives, and make explicit 
the role ()f time and its dynamic implications. 

It hoped that additional users of this proposed MCCM will 
continue to benefit from grmip^work reinforcing its use as an 
educational tool aiid showing the advantages of a simulation model as 
an evaluation instrument — characteristics that today are- perceived 
only on a speculative basis. 
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Table 1. Projected Government Expenditures on Health According to Two 
Health Policy Designs* — €k>lombia (billions of pesos at constant .prices of 
1970) 







Base 


Alternative 


1964 




1.5 


K6 ^ 


* 1975 




6.3 


6.8 


1985 




.13.2 


14.7 


1995 




27.8 


31.7 


2002 




• 491 


5^.7 



* See definitions in text^ . 

SouRCu: CCRR Estudio de Salud. November 8. 1975 • jl 

CCRP. Eftudio de Reforma Tribuuria. November 9. 1975 / 



AN EXAMPLE USING SERES HEALTH SUBMODEL 

As an example of the use of data generatedNby a simulation .model, 
some numerical results produced by the existing SERES Health 
Submodel (SHS) will be summarized and interpreted. The SHIS is 
uied here because of tlfe similarities of its structure to that of the 
proposed MCM; therefore, this example only partly reflects the use 
of a model such as the MCCM. 

According to the criteria presented in this document the SHS 
indices may be taken as evaluation standards. They may help in the 
diagnosis of healt^i needs, in the definition of priorities for the 
country, and in the statement of alternative program designs. 

The example is fprmulated in terms of two alternative policy 
designs. The base policy assumes that public health expenditure 
follows the historical pattern observed in Colombia during the last 15 
years. The alternative policy assumes an increase in the trend of 
public health , expenditures. The alternative was obtained by simulat- 
ing the increase of Government income that would follow a tax 
reform such as the one approved in Colombia in 1974. Table 1 shows 
Government expenses for health for each of the alternative policies. 

. The example compares the indices that result ff^m the two designs, 
and in this way it can give an idea of the described uses of the 
MCCM. * 

The indices included in ihe example refer to the achievements, 
effects, and impact subjects of evaluation; all of them are effectiveness 
standards. (Efficiency indices could also be easily constructed by 
combining the elements in which effectiveness indicators are based-) 
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TaWe 2. Child Clinical Morbidity Index, for MCCP,^ According to Two 
' Health Policy Designs — Colombia (thousands of cases) 







Ag?!$0-l 




Ages 1-4 




Ages 5-14 




Base 


Alternative 


Base 


Alternative 


Base 


Alternative 


1964 


2,135 


2,135 


5.361 


5.361 


2,578 


2,5781^ 


1975 


2,278 


2.278 


5.537 


5.537 


3,530 


3.580 , 


1983 


2,318 


2,292 


5,539 


5.501 


4,837 


4.832 


1995 


2,110 


2,041- 


4,970 


4.836 


6,355 


6.273' 


2002 


1,962 


1,936 1 


4.310 


4,242 


7,204 


7.000 



Sources: CCRP. Estudio de Salud, Nqvembcr 8. 1975 

CCRP, Estudio de Reforms Tribuuria. November 9. 1975 



Health Status Indices 

As indices, of the population health status and of cultural determi- 
nants, the SHS presents the sizes of different types of demand: 
biological (clinical morbidity), felt, and real. Table 2 indicates the 
biological demand and clearly shows the convenience of working with 
cases rather than with persons, since one person ca» become several 
cases with the same sickness during a given year. Froni 1964 to 2000^ 
the infant population under age 4 increases, but the biological 
demand does not.* Basically this trend appears because of changes in 
the country's developnrtent and because of epidemiological transition. 
Thus, if the public health organization assumes that the need for 
services wilUncrease in direct proportion to population growth, it 
may incorrectly direct its resources. Data also show that because of 
changing demographic structures and death rates, the country must 
be prepared to treat an ever-increasing number of cases between ages 
'5 and 14. It must be noted that the decrease (with respect lo base 
policy) in biological-demand cases from 1985 on, when the Govern- 
ment augments its public health education and family /planning 
budget ("alternative" column in table 2), is due as much to greater 
coverage of health programs as to a lowered birth rate brought about 
by fertility control programs (family planning and education). 

Indices of Achievement 

The achievement indices generated by the model are the treated 
cases and th^ physical and human resources they use. Treated 
children are divided by age into three groups (under 1, between ages 

• This result assumes a decline in fertility slower than the one found during the last 
intercensal period for Colombia (1964 to 1973). . 
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Table 3. Child Health Achievement Index (Effective Demand), for MCCP, 
According to Two Health Policy Designs — Colombia (thousands of treated 
cases) 



Ages 0-1 Ages 1-4 Ages 5-14 

Base Alternative Base Alternative Base Alternative 



1964 


1.309 


1.309 


2.089 


2.289 


238 


315 


1975 


1.992 


2.061 


4.616 


4.789 


' 2.577 


2.663 


1985 f 


2.249 


2.283 


5.161 


5.418 


4.068 


4.229 


1995/ 


2.106 


V 2.041 
^ 1.936 


4.949 


4.836 


5.916 


6.148 


2002 


1.962 


4.310 


4^.242 


1^^. 


7.000 



SouKCKs: CCRP. Estudio de Salud. November 8. 1975 

CCRP. Estudio de Reforma Tribuuria. November 9. 1975 



1 and 4, and between ages 5 and 14); these categories correspond to 
those used by the Mother-Child Division of the Colombian Hea^lth 
Ministry (table 3). Note that with the increase in public health 
expanses,. the number of treated cases grows (relative to the base 
alternative) in the early period, but diminishes toward ^e end of the 
century. Considering this trend, the need to tinderstand the above- 
mentioned relationships between health, demographic change, and 
the determinants of birth rate changes is vital. In the early part of 
the period the percentage increase in treated cases is lower than the 
increase in expenses (table 1), reflecting an increase \xs average coits 
due to the enlargement of health facilities and to changes in the 
proportons of infectious and degenerative diseases. 

In the model, physical and human resources are calculated from 
the number of cases treated and from a description of the amount of 
resources required per case treated. It is evidgnt, then, that the 
measurement of resources per case is crucial inBBkimating yme total 
resources utilized by the health system. The importance of preparing 
an accurate description of the observable structures' in the country 
afso becomes evident. It is possible in. the model to simulate reality as 
well as to incorporate alternative descriptions of the quantity of 
resources needed per case in accordance with technical norms. These 
norms could be fixed by a user of the model, permitting the study of 
the effects of changes in^ resources required due to changes in health 
service technologies. 

Indices of Effect ' 

The two major effect indices produced by the model refer to death 
rates and the duration of diseases. The model specifies that well- 
treated, cases present lower mortality rates lhan poorjy treated or. 
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Tabic 4. Mortality Child Program Effect Index, for MCCP, According to 
Two Health Policy Design.!^ Colombia (thousands of deaths) 





Base 


Ages 0-1 

Alternative 


Base 


Ages 1-4 

Alternative 


Base 


Ages 5-14 

Alternative 


1964 


85.0 


85.0 


45.0 


450 


14,0 


14.0 


1975 


78.3 


77.2 


,42.0 


41.4 


15.0 


14.9 


1985 


75.0 


73.1 


41.0 


38.7 


19.0 


18.7 


1995 


66,8. 


64.0 


34.9 


33.9 


23.2 


22.5 


2002 


59i^ 


58.9 


30.0 


^ 29.5 


25.1 


24.4 



SoumcEs: CCRP.' Euudb de Salud, November 8, 1975 

CCRP, Estudb de Refonna TribuUria. November 9. 1975 



utitreated xases; thus it is expected that a larger health budget and 
more treated cases will decrease the number of deaths. In table 4» 
death data (classified into three groups) are presented, and the 
changes resulting in these groups when Government health funds 
increase are also «hown. - ' . 

The differing behavior of deaths in the three age groups during : 
the period 1964-2000 (that is the^decrease in the first two and the 
increase in the last) is produced by fertility changes in the model and 
by the specification of the function about epidemiological transition^ 
The nuniber of deaths in the 0 to 1 age group de^;reases from 34 
percent of total deaths in J^64 to 18 percent in the year 2002. 

However, the statistics of the I>epartment of National , Planning 
(DNP) show that this percentage was between 28 and 32 in 1968. The 
differences between the DNP statistics and the results shown in table 
4 must be studied further, taking into consideration special note of 
the large? underestimation of death rates, especially for infants, in 
official statistics. 

The number of days of disab^ity shown in table 5 could provide a 
basis for estimating the effects of child morbidity on the available 
time of parents and on the performance of their children in school 
(for the 5 to 14 age group). 

Indices of Ihipact 

The quantity of health services and their distribution among 
population groups affect iM structure, size, and rate of growth of 
the population through changes in mortality and fertility. Thus, 
figures such as those on the size of the child population and the total 
population which appear in table 6 are examples of impact indices of 
health services. However, the differences in the demographic effects 
of the base and alternative policies are caused both by changes in 
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Table 5. Disability Child Program Effect Index:, for MqcP, According to 
Two Health Policy Designs— Colombia (millions of days) 





Base 


Ages 0-1 

Alternative 


Base 


Ages 1-4 

Alternative 


Base 


Ages 5-14 

Alternative 


1964 
1975 
1985 
1995 
2002 


16.7 
16.6 
17.0 
16.0 
15.7 


^6.4 
16.3 
1j6.7 
15.6 
15.5 


41.4 
37.6 
37.1 
^3.5 
29.6 


41.0 
37.1 
36.3 
32.4 
29.2 


32.4 
38.1 
50.9 
66.2 


^.0 
37.9 
50.6' 
(54.6 
72.1 



Sources: CCRP. Escudio de Salud, November 8, 1975 

CCRP. Ettudio de Reforma Tributaria, November 9, 1975 ^ 



public expenditures for health programs and by changes in futnUy 
planning and education activities, which also have impacts upon the 
entire >QpUlation. Therefore the measured impact is^ not exclusively 
due to health programs. It is possible to isolate the impact produced 
by additional health services, but this calculation would require 
additional experiments with the MCCM. We expect these to be 
carried out/in the near future. 

APPENDD( 

In recent years the interrelationships betw^jMN population growijfa 
and economic development have been a topic of Jively discussion 
controversy throughout the world. Analysis pfjtiq jjroblem is difficult 
because of the complexity of the issue. I^efote: attempting any 
analysis, it is necessary to develop a conceptual framework which 
describes the interrelationships between demogr^hic and economic 
processes. In a developing country like Colomb(^ ^> magnitude of 
these relationships is affected by Goveriinient p<^ies, particularly in 
the fields of education, health, and Tamil^ planning, * 

SERES is a computer simulatjton^ model >^hic^ is designed to 
provide a convenient way o^^tj^fing' these isjMyn Colombia. Its 
purpose is to analyze, by meqn^f|iutet:^4v a* 
the effects of changes in GoViSlf^mc^^ in the 

areas of education, health, famiUy jMn^^ overhead invest- 

ment, and taxes) upon the ecoii6jfnit^nd-<fthio 
of Colombia. - - » ^ .^V-'* ; * ; 

The fact that the model has b^Nck 'estii^a^f^^^^^^^^ d^ta 
imposes unavoidable Hmiutions, iD^ci^^ ofe t^^^^^ 

bility of historical information in a yev^ioping fountr^^^^^ is 
divided into seven submodels: goVernmjbi^t,-^^^^e^^^^ 
health,^ family planning, migration, and d^mpgpra^ is 
designed to make each of the submodel^va5.ijodei^ as possible. 
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As a result, there are very few interrelationships among the sub- 
models. ■• ^ 

The Government submodel, based on general policy objectives and 
on previous model output, determines the official economic policy 
for each year: revenue by category and expenses by program. The 
economic submodel computes economic activity using information on 
•Government policies, demographic processes, education, and eco- 
nomic activity in previous years. The health, education, and family 
planning submodels compute the educational structure of the popu- 
lation and calculate the number of deaths and the births prevented. 
The demographic submodel (using the educational strjjcture of 
women of fertile age, the number of births averted, and the number 
of deaths) computes the age and sex structure of the population after 
migration. 

A nontechnical description of the structure and uses ofthe model 
may be found in CCRP (1974). A complete technical description of 
the structure, data estimation, and validation of SERES, may be fourtd 
in CCRP (1975 a.b.c,d). 

SERES was developed at the CCRP in Bogota by a team consisting 
of Rodolfo Heredia, Manuel Ramirez. Bernardo Kugler, and Jaime 
Arias. This team was assisted by Alejandro Vivas. Alvaro Rodriguez, 
Franca Casazza, Edgar Baldion. and Isaias Roa. Programing was done 
by Alfonso Gonzalez and Alfreds Gonzalez with the help of Guil- 
lermo Rojas and Alvaro Lopez. SERES was developed under a 
contract with the Population Division of General Electric TEMPO in 
Washington. D.C. • 
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DISCUSSION 



David Bersh and Carbs V. Serrano 

By means of a simulation model, the Regional Population Center 
Corporation, in collaboration with the Ministry of Health and other 
institutions in Bogota> Colombia, is attempting to develop evaluation 
procedures to measure the impact -^nd effectiveness of a comprehen- 
sive maternakinfant health program at the national level, as well as 
to measure the limiting role of other factors on the. health sector. 

This method of evaluatipn is distinct from the others presented 
and discussed at this meeting in that it constitutes a method of long- 
range prediction and projection for national health planning. Perhaps 
its principal valu^e lies in- its imposition of a formal structure on the 
system: Key elements are identified and their interrelationships 
recognized and evaluated. Unlike other more restricted models and 
methods of evaluation, it takes a comprehensive approach which 
considers the system being evaluated in its totality, not as an isolated 
element. 

The groups considered the method most applicable to policy 
planning and formulation, given its ability to analyze and project the 
course of events. They stressed the program's value as an educational 
fool for the project staff. They agreed as well that the method has 
move applications as a predictive tool. One possible application is to 
start from a demographic structure so that predictions can be made 
about which population groups will be- at high ^k. Potential demand 
^ can be calculated for the at-risk groups; perceived demand can be 
' determined by the morbidity levels which are encountered. or calcu- 
lated using potential demand as a base. The response of the 
population to perceived demand will define real demand, and the 
options for in^proving coverage and prevention will be multiple if the 
required data are obtained. 

Several major problem areas were identified. First, the method 
involves many elements, and data have to be obtained from multiple 
sources of varying quality. Many gaps undoubtedly will be encoun- 
tered, since data for independent variables will be missing in many 
instances. The data will be soft, and calculations will depend heavily 
on estimates and approximations, and at times on old data (for 
example, the 1965 National Morbidity Survey in Colombia). The 
successful operation of the .model requires that all the data be 
ot)tained and that there be no major changes in national health status 
or health policy. All these problems are common xo simulation 
models, which originated and were successfully applied in fields such 
as industry, where data are hard, soft, or nonexistent, and where 
conditions are unforeseen. 
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Secondly, it was pointed out that highly trained professionals are. 
needed to adequately manipulate such a system or model and that, at 
a progprammatic level, personnel are needed who can understand and 
incorporate the results into health policy and programs. This type of 
human resource is not always available in less-developed countries. 
The application of a single-faceted model to a country where 
conditions are heterogeneous represents a problem in itself. 

Finally, the most important problem foreseen in applying the 
model is that, because of its characteristics, it may elicit considerable 
resistance on the part of decisionmakers and policy formulators, since 
such people frequently prefer simpler, more direct, and more rapid 
evaluatipn mechanisms. Similarly,' mathematical simulation models 
haye not been received with the desired enthusiasm in political circles. 
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EVALUATION OF CHILD CARE IN PRIMOPS: 
A COMPREHENSIVE HEALTH PROGRAM IN CAU 

CdLOMBL\ I 

Presented by Cildardo Agudelo Gil * 

PRIMOPS* is a demonstration program that is a coordinated effort 
of the Ministry of Health of Colombia, the Health Department of the 
city of Cali, and UmVersidad del Valle. The city Health Department 
ha^ overall responsibility for services and program operation, and the 
University performs a technical assistance role. A previous demon- 
stration in Candelaria, a semirural area, served as the basic model. 

PRIMOPS has the following goals: 

L To demonstrate an innovative system for the provision, of 
efficient, effective services which can be adapted and replicated 
at the regional 'and national levels. To provide such services, the 
system must fulfill the following requirements: 

a. Low cost 

b. Achievement of a high degree of coverage of the population 

c. Reassignment of an ever-increasing number of functions to 
nonprofessional and auxJ^ry personnel; 

2. To improve health conditions in the demonstration area, signif- 
icantly reducing current rates of- mortality, morbidity, and 
fertility, especially in the most vulnerable group of the popula- 
^. tion; 

S. To prepare, train, and utilize professional and nonprofessional 
personnel capable of more realistically facing the health prob- 
lems of the area; 

4. To carry out studies and develop operational methodologies 
which permit the evaluation oPthe efficiency of the system and 



* As the paper presented at the conference had many tables and an extiensive text, a 
, summarized version has Ijeen prepaid for this volume by the editor, in cooperation 
with the author. ~ * ''^ 

■ Spanish acronym for "Research Pro^^m in Models (be Proy^ing Health Services." 
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impact oh the heahh of the community and which offer data 
for the determination of jdter natives to the solution of healthy 

problems. 

-' , # _ ' 

Characteristies of the Target Population 

■ ^ ■ 

The target area is a sector of the city of Cali, which has a total 
population of approximately 60,000 living in five "barrios." The area 
and living conditi6ns of the inhabitants are considered characteristic 
of marginal urban areas of the region. In 1973, 43 percent of the 
families had a monthly . income that varied between $22 and $44; 
unemployment was approximately 18 percent;' and illiteracy was 30 
percent. The main target groups in the program are women of 
childbearing age and children under 5, who were considered the 
most vulnerable. It was found .that 38 percent of the children in the 
target area suffered from 

' Service Prindples-and Obj 

In order to achieve the program goals, the service model was 
designed with the following operational principles:. 

' , ' . ■ i . ■ ■. • . 

1. Coverage of more than 80 percent of the population, particu- 
larly of moihers and children; 

2. Ability to function within^a s;^stem of urban regionalization with 
a high degree of decentralization; * 

3' Satisfaction of the- following requirements: 

' a. Maintenance of permanent contact with the family uniK 
through home visits; 

b. Fulfillment of real and felt needs of the community; 

c. Maintenance of both centrifugal and centripetal activity; in 
dther words, provision of care to patients visiting health 
agencies as well as to people in their own homes; 

d. Maximum use of resources in eacK community, with com- 
munity participation in the planning and solution of- its 
problems; - - 

e. Use of professional personnel (physicians, nurses, and den- 
tists) with maximum efficiency, giving them more important 
responsibilities and more supervisory .functions^ 

f. Optimal use of auxiliary personnel through the reassignment 
of functions' while redefi:l:iing the role of such personnel in 
health programs; ' --^ % 

. ,/.g. Ease* of replication in other communities; 
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h. Limitation of costs to the^ budgetary allowance of the health 
, centers; 

i. Periodic evaluation of the system and its impact on the 
community. 

* The following scsvice objectives were established: 

1. For the mother 

a. Decrease in maternal mortality; 

b. Decrease in frcijuency and severity of: ' 

— Obstetrical hemorrhage, including abortion \ 
' y — Toxemia during pregnancy , 

— Obst^ricar(amniotic) arid puerperal infection ^ 
' 2. For the cKild 

a. Decrease in^fantile mortality; 

b. Decrea^ in ^llfrinatal nK)rtality; 

c. Decrea^ in mortality in preschool children (ages 1 ip 4); 

d. Decn^ase in frequency and severity of; 

-r- Eye infection and umbilical infection in newborn babies 

— Prematurity * 

— Diarrhea in children under age 5 

— Malnutrition in children under age 5 

3. For the couple » 'w-/^ 

a. Education and motivation ift responsible pSrentTiood; 

b. Provision and supervision of contraceptive techniques; 

c. treatment of infertility; 

4. For the community— education and motivation in the utUization 
of services offered". 



Evaluation Activities ^ ' , 

The external evaluation. of the program is earned out through the 
implementation ; of ;the following six research studies and the devel- 
opment of area profiles for both the target area arid a control group 
area^ with inter nat;ional technical assistance coordinated by Tulane 
University. . * . 

1. In the First study, a control area* similar insofar as possible to 
the demonstration area^Avas selected irfCali; and sampling ^techniques 
and cross section samples to be used in comparii^g the two areas were 
established. The cor^^ area was selected on the basis of geographic 
characteristics c^d incCime, racial Composition, and number of inhab- 
itants' The sample^ will provide data for comparison, of the target 
and control barrios jn 1975 (before the PRIMOPS intervention) and 
agajln in 1977 (afj^r). The community profiles will provide detailed 
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" • ' # . * ^ 

descriptions of the ijirgct and ^ontiHl ai^as and will identify favorable 

community cQnditions worthy of being stressed in the replication 
phase of,^the progi^m. background information gathered in these 
♦studi^ will bc^elpful in ui^cflfctancling the reasons for possibje 
unfcxp^tfd program reaul^J. 

2. !fc the sejC#nd^^udy. the ^gree of the ccmiin.unity*s acceptance 
of rejection of the proffcfpd health services is being meaisured, and 
kn<)|pfcdge, attjjudcs. and practices (KAP) are being assessed over- 
time. "F<?l4," .morbidity and h^^lth service utilization are also being 
survcyeS. OveralK the study attempts to measure the degree of 
consumer satisfaction vjith tlie health servi^s offered by PRIMOPS 
or J)y the, traditional systen\. Yhis^will be a ci'ilcial element in deciding 
the merits of flg[>lication of PRINfl^b^. 

t 3. The third study* measures, ^veif time, changes in morbidity, 
mortality, anrf fertility in the tdVget and control areas. It is believed 
that this study will eventually validate the assumption that improving 
the efficiency of tnfe official public health services in poor communi- 
ties docs indeed result in iirfpro^ed health and survival as well as in 
decreased fertility. This is a worthwhile effort, since few studies have 
documented the real impact df health services on health status.^ 
Naturally, it takes ii^to account concurrent improvements in water 
supply, sanitation, and ' housing condition^ included in the area 
profiles, which complete the related environmental conditions neces- 
sai;y to establish cause-effect relationships. « 

4i Th^ fourth study assesses the adherence of service personnel to 
PRIMOPS norms and the quality of such services which, although 
heretofore, performed by physicians and nurses, have been delegated 
to bthirs lindfer PRIMOPS. One of the purposes of this study is to 
counter probable attacks on the quality of delegated health services. 

5; The fifth study measures or defines the roles and role structure 
established within PRIMOPS and tljip acceptance of PRIMOPS and of 
their own role? by the personnel of the project. It also' ''analyzes 
interpersonal relationships v^ithin the project. This study will provide 
useful inf<|^lrhiatkm for detect!^ personnel profelems and offers 
relatively rapid ^tedback to management for corrective modifications. 
The definition of role structure and of imerpenifHial relationships 
will provide important information for ix)ssible changes to improve 
the efficiency of PRIMOPyin its replication elsewhere. 
, 6. The last study is aimed at the establishment of^ reliable cost-, 
analysis procedures, allowing the measufement of cost-effectiveness 
and cost efficiency. Amgng the requirenients to be met by PRIMOPS, 
the provision of . low<ost health services is a most important one. 
Existing cost•acco^nting pfocedifres do not allow for the detailed 
analysis df cost data. It will'be necessary to design better ways, of 
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allocating costs (simultaneous equations, cost centerSt and so on), and 
consequently to be able to determine service costs by subprograms as 
well as by operational and development costs. Otherwise it will be 
very difficult to establish the feasibility of the replication phase. 

The PRIMOPS services have been financed entirely by the Colom- 
bian Ministry of Healthy based on its policy of strong support of 
regionalized services, decentralized administration, use of paramedi- 
cal personnel, priority for maternal and child health, and jextension 
of services to the poor. 
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DISCUSSION 

David Bersh and Carlos V, Serrano 

Before discussing evaluation, it is important to indicate that the 
Research Program in Health Services Delivery Models (PRIMOPS) is 
not exclusively an evaluation modpL Rather, it has a more general 
focus, of which evaluation is only one part. The primary objective of 
PftlMOPS, a joint effort of the Universidad del Valle.and the 
Ministry of fiealth, is to design, implement, and evaluate a model for 
comprehensive health services that can be replicated in other com- 
ihunities. The program', which until now has emphasijted maternal 
and infant health*, attempts to expand low-cost coverage, to^deliver all 
levels of health care on a regional basis, and to foster maximum, 
active community participation. 

In the course of the presentation and discussiops, it became very 
clear that PjftlMOPS is not in itself a pilot project, but a program of 
experimentation and demonstration of normative models and meth- 
ods 6f evaluation. The program is expensive to operate, but within 
the context of a comprehensive system, the experience and outcome 
can be worthwhile. In the case of Colombia, the National Health 
V System will, addict, refine, and replicate th^|jnost successful program 
elements oh a broad national level. In.otherVords, under the proper 
ciiTumstances, the program and its methods need not be reproduced 
in their entirety to have a wide impact. 

In^jthe course of the discussions, several limitations, a5ytd«*4rom cost, 
were identified. PRIMOPS is, for example, an urban program 
oriented toward a poor population. The modifications needed for its 
application to other types of urban populations or to rural popula- 
tions mjjst be identified. Also, the number of variables studied is so 
large that their satisfactory control appears difficult. As a result, large 
quantities of data must be gathered; this situation in turn causes data 
collection problems. Additionally, there is a possible dispz^fity in the 
quality of the data for the study group and those for the control 
group. For example, the greater vigilance and tneticulousness exer- 
cised in collecting morbidity data for the study group liiay rhean that 
relative morbidity for this group appears higher than it actually is. 
Control group data also present certain problems in comparability. 
' Although it is not easy to mtiintafti equilibrium between the 
scientific and technical aspects of th^e program and its evaluation 
componen>^ the political copsiderations and community e?^jf)ectations, 
this goal should be pursued. It was felt that without this balance, the 
occurrence of disturbing situations \^s possible. For instance, if the 
Government were interested in quicks results and did not allow 
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sufficient time for evaluation, the academic world would want t9 so 
amplify the theoretical implications of the program outcomes that 
results would be extremely difficult to implement because of their 
complexity or untintelincss. 

The program evaluation model was, designed with solid technical 
assistance and was carefully conceived and tested. Although the 
difficulty in obtaining the required data and the desiriibility of having 
appropriate preexisting sources of information was noted, it was felt 
that the program could be replicated in other cultures. Also, despite 
the fact that the proposed methods are expensive, it was recognized 
that after several applicatioijs, it would be possible to select the most 
useful indicators and procedures and so reduce the cost to approxi- 
.mately 5 percent of the total budget. The program as a whole already 
occupies a recognized place in the national and international arenas. 
It has begun to pfoduce results that demonstrate that^ the bcnefits^ 
will significantly exceed expectations and will justify whatever initial 
investment was required. 
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GOAL GONFUCrS IN A CHILD HEALTH STATION: 
A PROBLEM IN ACTION RESEARCH 

Jane Z. Moss, Renate Belville, Bernard Levenson 

Introduction 

' Th<& objective of the research described here was to make recom- 
.pniendatioiis for^the sol\itioD pf an organizational problerA in a child 
/hcahh phjjject. ;Th^. the unpredictable fluctuations in 

rd<nnand that aiFfi^^^ of the staff to perform according to 

' the g^^ls of the prqfj<tct. Thisj&sue was one in a series of organiza- 
tional problems wHfr which, tKe authors worked during a period of 
"concurrent 'ipva1ilati^" (MoM^^^^ in the Wagner Child Health 

Project." ■ 'i? ?^",^C><:^V ' "'^^^ 

Concurrent evaliKftion is/iased on.the pVJnciples'bf policy research. 
The distinction and justification for Jipjfcy'* Research has been clearly 
stated by Coleman ( 1972. p. 3) as followsJ * ' " 

... policy research bridges two' worlds with very different properties: 
the world of the academic discipline and the world of policy and 
action. 

The defining characteristics of policy research are two: the 
research problem originates outside the discipline, in jhe world of 
action; and the research results are destined for the world of action, 
outside the discipline. The special properties of policy research stem . 
f^pm the different properties of the disciplinary world and the 
world of action, and from the translation problems involved in 
moving between these two worlds. 

Using the rpethods and concepts of social science, concurrent 
evaluation conducts policy research within an organisational setting. 
The social scientists translate an organizsitional problem into a 
research problem. Findings from the analysis of the research problem, 
are presented to the executive sfaff with recommendations for 
change. The executive^staff can only accept recommendations which 
it considers realistic ancf ari^improvement to the organization. Conse- 
quently, concurrent evaluation must be cognizant of the options and 
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constraints of the organizational staff and setting. T|i^ k^^ 

from the academic disciphne is used to inform A.hc M^Pc^^ of Tictlpn 

about the nature of the issues and the realistijt^bpUo'ria ^vailaMCi^'^ / 

Once implemented, concurrent evaluation' :h«?sV^ :buHl'in^^^^^^^ 
and-after research design. Data generated, ffoiri the^'V^ 
decision car^bc compared with data used as.:a basii fpr , the ;r^p 
dation.* The process can be repeated as th<implcmenle4; decrisw^ 
modified. Policy research must also b,e';qo^)iJz^i]ft af^ 
organization of the staff when recommendations ife njacle; The s 
that implements change must integrate ih^at chan^^^ 
configurations. This change usually requires ^b^^ from the 

research team. / . , 

The concurrent evaluation approaahj has^eVOiV^^^^^ 
steps for the research process within .tiie;oi%artlzatior^':^ 

— Assessing the organizational sti^ctu^,-Qr A with 
respect to the goals ofiithe org^nizatibni' i^^Clu^i tec^efinitipn or 
Nspecification'of goals Where they .^re iiiapprppriate ot vague; , 

— Setting pnorities for th^ orgahizatiop^l ^prpbletfts located in the 
assessment; ^ . / ^ ' / •*» .V^ ^ : * . ' • ' * - '* . 

— Designing and rmplen)en>ting. . research! based on the 

problems th'at'have been ranged; ^ . ! ' . • : v >• ' ; • 

— Analyzing Yesearch /indi!]igs ^nd puttltig fh^ ittipl^^^ of such 
findings in. tlie ^form^bf recommendatiphv fp^. those persons wkh 
authority to Accept or reject thein; ; ; .-^^ * • i . 

— Covinselirtg the staff in the impfJemeritatiPn of, accepted recomnien- 
-■ .dations; ^ , ' . ^ ''y'.''>- '^^ '-''-'l 

— Reassessing the^organizatiorial'StrUciijire;^^ ^^^^^ - 

— Reviewing che next organizatiPtral pToU^irt appiro for re- 

search;* • ■ ' * ■■. '/.■■:;■*■■/■ ' 

This ^approacrt emphasizes the ihts^grative ^pm involved in the 
application of research findings and the u^e bf.i^xploratory statistics^ 
with the most appropriately -ay^i able 'or easily collectible data: It 
assumes, th^t any service is a dynamic "pro^ and not a static 
operatioa. , As a : process, change$ ihtrpiduced into one part of the 
orgpanization produce; a ripple effeqt in dther paru, of the organiza- 
tion. Until these effects hav^,been'integrated into tlie^s and the 

a '. .v '-:, ■■; ; ' ,'' ' ' 

* CampbeU states: ... thiis ccKC^riocmists^. operatipris ^ and mathematical 

dc^sicxn theorists trustingly extrapoUlte ^ixini past science and conjecture, but in 
getieral fail to use the intjilemcnt<^d depsiohi to torreci or expand that 'knowledge 
■(•1971).** ^ -. ■-■^ ;^:^;^■/.■^/•^: . 

, * The approach to exploratorY, statistics is* explicated in the work of John Tukey 

(1970). ;■ ' ■;■.•> * ■• , ' 
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policy outcome determined, continuous monitoring for reassessment 
and realignment is required. Consultation with the responsible staff 
is continuous and detailed. 

Resolution of problems according to a priority of organizational 
needs leads to a sequential series of research questions, explorations, 
analyses, recommendations, and consultation — before, during, and 
after implementation. The problem reported in this paper is one 
research question in such a series. The staff of the child health station ' 
had already resolved a number of structural problems^ with new role 
definitions, new functiofis identified within the child health station, 
and new designs for data collection. At this point, the' staff members 
were struggling to keep up with their own work plan. The unpredict- 
able fluctuations in demand for the services of the station^ affected 5: 
their ability to perform according to the goals they had set for > 
th em scjves in the project. 

Defining the Project 

^ The goal of th]e, exj/eritnfcrit^rf)roject in the Wagner Child Health 
Station was to tiri^at'thitf thild Jh the context of family and neighbor- 
hood.^ This project had three objectives: 

— To provide curative and preventive care to all children eligible to 
visit, the station, by age and resijdence in the health area; , - ^ 

— To educate and train parents in the health care of their children; 

— To contain costs afttr t»he developmental phase of the prpjctt to 
make replicatio*\:in existing .health agencies feasible. , ,^ 

' •• -i.-' .•■■.^ aV' ■ ; ' ^ - ' . . . 

The codirectors the special- p^roject at Wagner were a pediatrician " 
and a nurse-clinician who hati,.*selected the Wagner station because it 
was a deprived area with no health services in the immediate vicinity. 
Confronted with the variety of problemis for the publicatiori^nd a 
working knowledge of the strengths and limitations of their own 
professional traditions, the codirectors realized the need for a new 
approach between the providers and consumers of health services. 

The new concepts for service were reflected by a greater number 
of clinic sessions than the station had previously held as a preventive 
care service and by new staffing at thle .station.^ It included an 
expanded role for the nurse as nurse-practitioner . and primary 



^ The project was developed by the Department of Communily -Medicine at the 
Mount Sinai Medical Center thrbygh'an agreement with the New York City Health 
Department. The codirectors of the projec(^ were Nicholas Cunningham, M.D.. and 
Beatrice Thomstad» R.N. , P.N. A. 

* For discussion oi these new staffing concepts, see Thomstad et al. 1975. 
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caretaker, a restricted role for the physician as a consultant specialist, 
and new roles for community health workers as assistants to the 
nurses and as outreach workers., What developed from the experience 
was a new type of service geared toward a specific patient population 
that had its own social and cultural definitions of health, illness, and 
social problem solving. ' 

Th^s'e families had not one but many problems at any given time. 
Problems with housing, landlords, unemployment, crime, and school 
dropouts, as well as those involving family ajid health, all competed 
for attention. Family resouf^s for solving these problems were 
limited. Finantial means were minimal. Agencies designed to provide 
help and services w,ere distant, cumbersome, impersonal, and over-, 
loaded with wprk. taced with many problems and few resources, the 
parents tencftfS Jo respond to problems orily when they erupted- into 
crises. Crises determined the priorities. * ^ \ 

The health teams offered help, advice, and services -fdr any 
problem which the parents brought to their . attention. The staff- 
understood that it was necessary to respond to the imrnediate: 
priorities of the parents.* The resolution of higher-priority 'crises;' 
permitted parents to give attention to health care it) turnvThe 
acceptance of parentally defined priorities established communication 
bridges bc?tween staff and parents. This bridging of the communica- 
tion gap with the parents worked so well that parents and patients 
came to the child health-stado" large numbers. 

The staff of the station was caught in an organizational dilemma. 
The success of the educational outreach goal brought about increasecl 
accessibility, to the station/ This heightened accessibility, in turn, 
resulted in increased demand in both the number and duraMibn of 
patient visits. The number of patients presenting themselves for acute 
care services fluctuated widely from one day to the next. The;(lme 
required for response to the number of patients was sabo\aging 
further educational achievemefTt and thorough preventive work. This 
crisis orientation of the population had interfered with" the planning 
of the staff^embers and fbrced them to adipt a crisis response. The 
staff problem was translated into a problem for the research team 

' \ - ■ y 

The Research Problem 

How could we account for the fluctuations in patient demand? If 
we could account for them, would the crucial variables lend them- 
selves to manipulation for planning? If th'ey did^ would we then be 



* Maslow (1954) has emphasized the importance of the ranking of needs by the. 
individual. 
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able t© smooth out the day-to-day p,uctuations in demand so the s?aff 
could plan its activities in a rational, roanner, reducing the emergency 
atmosphere which tended to eliminate preventive and educational 
priorities? 

If we could jjolve the problem of the fluctuations, then the "staff 
members might be able to preserve each of the goals they had set f or 
the station in their integrated model of health care delivery. If we 
could' not, then they would have to make a decision about setting 
priorities for the health station, making a conscious choice concerning 
which goals to eliminate or reduce, A very real alternative would have 
been to turn the child health stafion into a children's emergency 
room emphasizing acute care to tne exclusion of preventive and 
educational service goals. Inherently, i^e latter course of action would 
have changed the character of the sta 

Establisliing Components of Patient 

The xiata initially available were daily .^pointment sheets kept by 
the receptionist for the station. Appointment sheets classified ibe tyjpe 
of visit: scheduled prevennve visit,' scheduled follow-up vlijtts^/ an 
walk-ins. Broken appointments were recorded - '* . ' ' 

We explored a 4-month period from January through April of the* 
current year as the first approach to understanding patient care 
demaijd. 

Patien^ demand included three types of services: 

— Acute care services; 

— Prevehtive'care services; 

— Educational or interpretive services.* . 

In bur present discussion of demand, 'we are concerned with the 
first two tyijes, which have different forms of expression by the 
patient population. Walk-in patients indicate the need for'acute care 
services. Patients with scheduled appointments generally indicate a 
need 'for preventive care services. 

O^r two indicators represented different types/ of variables for 
planning: controllable and uncontrolfable. For purposes of policy 
research, this distinction is important. Most of i^he scheduled appoint- 
ments were controllable by the staff. The walk-in appointments were 
uncontrollable. . 

Because it was a controllable variablei we examined scheduled visits 
first. The codirectors of the child health station had developed 
preventive protocols which defined the number of preventive visits 
•^.for each year of life (Cunningham et al. 1975). We assembled data oq 
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.Figure t;.. Visit formula 

, the number of children by age from the census and from Registration^ 
' information. These data permitted us to calculate the total number of 
preventive vi'iiits for the station on a yearly basis. Given the existing 
staff and the timl required for a preventive visit, We determined that 
the station would ne^d an average of 15 visits per working day to 
maintain its preventive services. -However, study qT the appointment 
sheets from the 4-month period revealed an overall 40 percent 
broken appointment rate. Consequently, it was necessary to]schedille 
21 visits per'ilay in order to realize an average of 15 actual Visits, v 

Acute care, by definition, was not a controllable variable. How 
many walk-ins could be expected? Based on our analysis of the first 4 
months, there wei*e. on the average, 14 walk-ins per day. 

Given the average figures for preventive visits and for walk-ins, 
follow-up demand could now be estimated. A-s described previously, 
follow-ups were generated from two sources. Follow-up appointments 



i6'i 



, . (}oal (^onHicts in a Child Health Station 1 59 

were given to 15 percent ofthe preventive visits and to 67 percent of 
the walk-ins. 

Based on these calculations^ a typical day — including, both preven- 
tive and acute visits — could be. constructed as shown in figure IT This 
first approximation based on 4-ttonth average .figures provided a 
rule of thumb for establishing different types of demand. The 
formula took into consideration those factors comnHoh to most texts 
on health planning (Rosenfeld 1971, Gentry 1973). It reviewed 
demand by age for preventive protocols and utilization characteristics 
by type of visit and included standards sety-by the station for quality 
care. Overall, the formula characterized the optimal combination of 
preventive and acute visits offered by the available staff. 

The formula was not intended as a realistic plan' for a visit quota, 
because it did not take into consideration any fluctuations in demand. 
These fluctuations, not the aggregate workload, interfered with 
planning for the delivery of services on a daily, weekly, or monthly 
basis. , 

Far (^oiu prescribing visit quotas, this formula was a first step in 
providing the planner with the components for manipulating de- 
mand..i*We hoped that understanding the interrelations of the 
demand variables would permit a fiexible response to the fluctuations 
in.;/iemand. To explore the fluctuations we decided to analyze a 
lohgei* timespan of utilization. 

Analysis of Demand Components 

We explored the daily appointment sheets for a 9-month petiod 
.from January through September, We began to compare the number 
of scheduled visits with the number of actual visits. Scheduled visits 
tx)nsisted of preventive and follow-up visits. Actual visits were both 
kept appointments and walk-ins (figure 2), Except for January^and 
February, sched tiled visits exceeded actual visits. If the volume of 
scheduled visits had actually materialized, the station would have 
bi^n heavily overloaded. In the beginning of the year, both scheduled 
and actual visits showed the largest number of patients. Scheduled 
visits continued to fluctuate around a mean* of 40 patients per day. 
Actual visits, however, exhibited a downward trend. The number of 
patients fell from a peak of 50 in January to a low of 25 in June and 
did pot start up again until August. 

Although variation in the scheduled visits showed less dispersion, it 
was more than we had expected. We had anticipated that, the 
schedulecf visits would be very evenly distributed throughout the 9- 
month period. However, if we keep in mind that scheduled visits 
consisted of preventive and follow-up visits, thep the variation is 
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Figure 2. ' Variation of JcWuIeii^ and aaiial.visits/'ln order to highlight the trend and reduce the day-to-day (luctuations, 
running medians wer^ used to calculate scheduled appointmctits and actual visits. The running rftedian is a moving average 
which is bt suited for smoothing variatibfis if the period selected matches the dp period! Based on thei-day cljnic week, 
running medians were established from 5-day peq^^^^ 
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better understood. Fr^m another study we had conducted,* we knew 
that 80 percent of^^e presenting conditions during January and 
February were diagnosed as upper respiratory infections or as otitis 
media. The acute (^jnditions during this 4>eriod were swelling the 
numbers of both walk-ins and scheduled follow-up appointments. 

The diffCTeijce between the scheduled visits and the actual visits 
was accounted for by two components — the number of brojcen 
appointments, and ^tbe number of walk-ins. If walk-tns equaled 
broken appointments, then scheduled- and actual visjjs would coin- 
cide. However, no correlatiori- between, broken apppintments «&nd 
walk-ins was fbund. - *. \ ' * ^ , 

Actual visits consisted of kept appointments and walk-insr Since 
broken appointments were uncorrelated with walk-ins, kept appdintr 
ments logically^ also had to be uncorrelated with walk-ins. Conse- 
^quentlf, we^wer^ able to look at these cornponents independently. 
Figures 3 and 4 presertt ihp nurtiber of h^pt appointments and the. 
number ^>f walk-ins respectively, plottetl fpr a 9-month period. ' / ^ 

Thrbughout the .9-month periQd4>kept'j appointments showecJ,,no 
seasonal differences and were, relatively stable. The number qf w^^k- 
ins, on the other hand, \Vas highly seasonal."^ he greatesfriuiflber of 
walk -ins occurred in the first months of the year. They decreased 
considerably, beginning with April and continuing through August, 
and rose again in- September. ^ 

Means and standard deviations were calculated for both kept 
appointments and wal^-ins. A comparison of their variability was 
more critical for plSnrting than a comparison of means. The variabil- 
ity Tor the walk-in* more than doubled that of kept appointments. 
Seasonal variations of the^walk-ins had a high of 17 visits in January 
and leveled off at 8 visits in July and August. In the same period,* the 
number of kept appointments fell only slightly, from 26 to 22. 
Variations from the monthly average decreased for walk-ins through- 
out the summer; in contrast, tl^ /monthly average for ^ept appoint- 
ments showed no significanj: variation. , < 

Although we cannot control wajk-in visits^ we can predict thena. 
They are higher . in the fall and winter months. Witff an ability to 
predict the uncontrollable variable, W£ can manipulate the controlla- 
ble variable of scheduled visits as a flexible, adaptation ^o th^ seaspnal 
fluctuations. • . . 

We recommended to the staff a reduction in the schedulin§[ of 
preventive care , appointments ^ufing the winter months, wioh a 



•The series of studies performed within an organization provide^ an a^|(;^niulated 
' knowledge base which (fiv|?s any one of the studies a broader context for inti^fpretaiipn 
and, insight/ * • ^ : ^ ■ ' ' ^ 
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Figure 3. Variation of kept appointments by month 

• i ^ 

gradual incrcase/in appointment scheduling from April litfough 
^August. We aruijfcipated that seasonally adjusting the preventive care' 
visiis to the acute care pattern should have made the visit volume 
smoother throughout the year. Even so. within the reduced and more 
manageable Volume, day-to-day fluctuations still remained unpredict- 
able.. . . . ^ 

Since we Ijad modified the demand side of the equation as far as 
possible, further improvement could come oinly from the staffing 
side. The staff had been divided into two tpms. each diHicted by a, 
nurse-practitioner as primary caretaker, assisted by%ree community 
health workers. (Tlie" physician was a consdltaigpt^ut not a Member 
of the team.) The t^ams alternated each w^e^ between working in 
the clinic and the field. During the field^week^^he nurse-^r 



was the coordinator of the outreac 
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Figure 4. Variation of walk-ins, by rffontlji^ 

BeaH^e the greatest prcssnre^rom patier^ueues in the ^lirAc was 
on the 1iurses> we recomnAidcd^that tl^ nurse onjhe field team be ^ 
comsidered M a routing backup oerson for the cl^ teariT. If there 
had been n<f concern with fisc^pifoblem^^ we mig|;it have recom- 
mended hiring an additional nurse. Howler, financial' cons^lf^nts 
required the use of existing staff onlyg ^ ^ 

In order not to^g|up^.^e field week, we recomnrfef^ded thafrteaips IJ, 
develop a more elaborate reporting sjjsl^m between community 
health workers and nurse-practitioners f Jt^ the fiejdwork. In this wj^iff, 
the fieldVvork woirtd continue, unhampered wln^n^fie nprse assigs^L 
to the fidd.had to serve as a b^fHup in the dinic, Techq[||ally difficulf 
cases w^^uld be reserved for the nu^ji^-practitibner. Routine field vi^ts 
would be delegated to the community health workers:' The develop- 
ment of a re^ed reporting system would initially ^i-eqpire additional 
training for the commuiiky Health workers anct considerable effort 
on the part of the nurse-practitioners. However, if effective, the 
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system for reporting and 'delegating would increase the probability of 
accomplishing the multiple objectiv^aof the station, At the same time. 
itSvould make the entire staff more aware and more articulate about 
its fidld week activities. 

The problem of supporting a multiple -goal program, including 
preventive and curative care with educational outreach, requires 
constant alertness on the part of the codirectors. The dilemma 
continues, but with somevyhat more control, when the problems of 
acute care demand are anticipated ar^d inobrporatecfc.into planning. 
We might add, as an epilogue, that the research team did an 
evaluation of the field w^ek activities in conjunction with the service 
staff. This evaluation .resulted in a i;pdefinition of Wlgh-risk patients 
aiid families' It also produced impressive evidence that the fieldwork 
was ^ highly effective component of the health services offered by 
the child health station. At a time when nationwide immunizatfon 
rates for preschool, children had dropped to an alarming low 
(German et al. 1976), 95'percent of the children at the Wagner 
station had been im^piunized for diphtheria," pertussis, tetanus, and 
polio. ^ ' 

■ . ■ * 

Sumnniary 

In this paper, we have detailed the organizationftil* problem, the 
relevant research, and the recommendations for application;© the 
problem: This organizational problem of the child health station 
stemnrfed from unpredictable, widely fluctuating daily demand for 
services. The unpredictability of the demand disrupted staff members 
in the organization of their work, thereby threatening lihe achieve- 

.Oient of their giultiple goals, particularly the preventive and educa- 
tional ones. In order ^o fope with the pco^tem of large and 
ilnpredictable fluctuations, the ^resear^^h team began a series of 

^ucces^e studies/each of which resolved some part of the problem 

in its application. * > 

* • ^ 

First, the j^esearth team investigated appointment sheets, available 
from th^dngoing operation of the station. Based on the analysis of 
the data, art optimal visit formula was^ established according to the 
requirements of tH^ staff and the dge-specific protocols for the 
station. • ♦ «? ^ * 

The next step in the research process involved a comparijdn by 
type of visit to study the denPianxJ for acute and preventive care over 
time. This analylls led to the recommendation that the staf^^ adjust 
the controllable variable, name^^j^reventive visits, around the sea- 
sonal fluctuations of the ullcontroll^'ble, but predictable variable of 
walk-ins. The effect of this shift in the planning of preventive 
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• . • 

appointments was intended to smooth out the extreme variations and 
kc<^ the total volume of visits more constant throughout the yei^f. It 
would reduce, but^ot eliminate, the daily fluctuations. 

Since we had exhausted avenues for manipulation of demand, any 
further response to fluctuations had to be made by manipulating staff 
assignments. Specific recomw^ndations for staff reassignment com- 
pleted this phase of the study. ^ 
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DISCUSSION i 

A 

Maurice Backett and Barbara Starfield \ 

Categorization of this paper according to the type of reseatch was 
difficult. **Operations research" or "efficiency analysis" seemed a 
more appropriate term\han "research" because the study was aimed 
more at obtaining information to solve a problem in a particular 
situation than at pi-oviding new insights or new knowledge. The 
essential difference between the two types of inatiiry is in the^xtent 
to vyhich the various factors impinging on thtf dependent variables 
are known. When the study design deals only With a limited number 
of dependent and explanatory variables, it is nc\t possible to generalize 
the findings because the degree of similarity of the facility in which 
the study is done to other facilities is not known.^ 

Although Ms. Belville stated that the work presented in the paper 
was only one aspect of the involvement of the researchers in this 
community health center, the relationship of this work to the other 
activities is not presented. This omission led several discussants to 
express concern that the narrow concentration on only one relation- 
ship (in this case, that between scheduiin^ of appointlnents and clinic 
workload) might lead to a failure to recognize other (perhaps 
undesirable) effects' of the intervention. In particular, there was 
concern that the objective of smoothing out marked variability , of 
patient load, although desirable for providers, hiight adversely affect 
patients by maiking their careless continuous. 

An advantage of this type of study is that implementation of 
research findings is optimal. T^ose who are involved in providing 
care are also involved in defining the problem and setting the 
objectives, and implementation of the findings in that setting should 
follow naturally and easily. U n fort u na tely,flef fee tiven ess of implemen- 
tation could not be examined by the researchers, since^the ^inic was 
discontinued because of financial exigencies. 

Applicability of the findings to other situations is n<^ possible, 
however, because the research was tailored to part4cVifar f^ovidefs^in 
a p^ticular setting. However, the applicability of the methb3 itself, 
whereby research is based upon observed needs, is pptenliaWy 
gencralizable. But even here, fransferability of the method would be 
limited in situations where financial support for researchers is lacking 
These researchers were fortunate in having indepejfident funding, 
wTiich made it possiblcTo'respond to research heeds as they arose. In 
most situations, definition, of a problem is followed by the need to 
apply for a grant to address it,, thus involving. a long lag time bd^tween • 
problem identification and research. , " 
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Thus, discussion of this paper showed how jea^y Ifris for researchers 
to trade generalizability for ease of implemeritation of findmgs (and. 
vice versa). The more generalizable a result is, ^he less likelihood, that 
it will lead to change in the setting" where it has been generated,, 
because the question may arise fjco'm the/perception of a wider, 
prohlem rather than from one perceived by the faciHty itself. 
Conversely, the less generalizable the result, the .more likely that it 
will produce- a change in the particular settling, bfecau^ it arises from 
within the- setting. Researchers should be able to respond to both 
goals. In order to do so, they must have 'the stability of financial 
sup{>ort to respond to needs as they arise in a particular setting and 
must plan their research so that all th^ impoftai^t variables are 
included in the design. 



/MJtRrnON INDICATORS AND THE BARBADOS 
, mSCHOOL GrtiLD NUTRITION INTERVENTION 
. : ' i*RCX;RAMME, 1967-76 ^ 

^ ; J ' . z^ - " ■ ■ , 

>vv, ' ■ . F. C. Ramsey 

.VStfinituuy" ' ! , 

' In 1965, the cKildreh's ward of the Queen Elizabeth iJ^tijipiter 
(QEH) in Barbados Vas faced w^Ji overcrowding, cross-inf^ction, arid ^ . x?*^'^' 

" staff shortages. T6-deterrnine the cause of the problem, ^40 cbnsec-^ 
litive admissions to the children's ward ugl^^ the autht>r's care were 
analyzed' in that y«t;^Thi^study.^)D the overcrowding was « 

due mainly to the^missioj^ crf^^^ diseases as.gastroen- 

tcntis a^<^^ife^aln^t^^n^ V ■ v^j^ . ^ ■ ; 

iir^igeef '^^^ tcfel a<to^i6ns-<^^ analyzed in order' to . '/A 

'^ dti^vmint^^^ |!(roblem. This analysis showed that 143 

caser (ifit)^ lp»ei*ce^| v^e energy^protein malmitrition >^ 

*:'t£PM) and th|t i|^e niajoriiif; >vere 'und^ i. The readmis^n rate.; ^ 
% P; with'^ was hi^hi at 25 percent. Jg^e mginagenlent of ^ 

; thc$t .fej^^ costly. We xctetfer^ined^fh'at 89 percent of . * ^ 

: tfejMe c^^ to fpllowup; fHSs' significant Cindingemph^- ; 

1? vsized^t^ ' r. - ' 

Injt^ryeQtioti PAgrahinie, enriWrading c<^iinuity oj^?^| 

. . care b^#ie^Q^^^^ and iht, cojfiynunity, jkvas introduced iC^^^^ 

• .1^6?- df:jthis pVogram was to reduce overcrowd ing^^*^ 

6n thie waril thix^^ 

f^Hospitalndays fo '^iJ 

---Case-£waUty^^ hospital? 
-^Pdinrijrey^^^^ EPM in the community. 



*Tlie; Mudy repoHcd hjjre wait^jupfiorlcd jn part by funds from the Macy 
Foundalioi^, ihtenViliiaws/^^l^^^ Research Corporation. OX- 

FAM, Milb^nk Mcrnori4t|K>nd. and;CAFOD.*. T 
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During the 10-year period from 1967-76, a marked decline was 
noted in: 

— Hospital-days, from 7,825 to 1,353; 

— Hospital readmission rate, from 25 percent (1966) to 0 percent 
(1976); 

— Case-fatality rate, from 13.6 percent to 0 percent (1976); 

— Community point prevalence rate, from 16.5 percent to 10 percent; 

— Mean ^^ily bed occupancy, from* 90 to 27 (in a 54- bed ward). 

The following indicators were considered the. most suitable for^ 
measuring changes in the nutritional status of the Barbadian pre- 
school child: the infant mortality rate, specific mortality rate for 
malnutrition, weight and height in relation to age, classification of 
children according to the extent of malnutrition — using Gomez and 
World Health Organization (WHO) classifications, percentage of 
readmissions, number of days spent in hospital by children recuper- 
ating from malnutrition (hospital-days), and the number of admis- 
sioft*-of malnourished children to the hospital. 

The utilizallion of results is discussed in relation to developing 
priorities, ^he establishment of staff positions, the development of an 
information system and road-to-health charts, the motivation of staff, 
the* conducting of nutrition surveys, and the establishnlfent of the 
Nutrition Centre of Barbados. Publications, training,j)r6gram expan- 
sion, expenditure, estimates, research, and invitauons; to scientific 
meetings are also considered. 

- In the discuilion of results, it is evident that the nutrition indicators 
demonstrated ^ marked improvement in the nutritional status of the 
pr e^fio ol cFiild during the period. However, the absence of a control 
groi|j^madc it impossible to carry out an effective, objective evalua- 
tion of this, nutrition intervention program. * 



Purpose ^, • 

This study, was initiated in 1966 by the.author in Barbados in order 
to create an appropriate, service-oriented nutrition intervention 
program that would, alleviate the conditions of overcrowding. (two or 
more infants in some cots) apd cross-infection on the QEH children's 
ward. These conditions were caused by the high admission rate and 
long duration of stay of malnourished children. 

The QEH is.ihe only Government-sponsored acute general hospital 
on the island..It« has ^00 beds — 54 in the medical pediatric ward and 
28 in the special baby unit. In 1967 the hospital established a teaching 
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-program for medical students in their final year. In 1976 the 
program was extended to studenis in their second clinical year. 

The need for the program wa^ clearly shown by the overcrowding 
.^on the ward (a mean daily bed occupancy of 90 children in a 54f.bed 
ward), in the outpatient clinics, and in the emergency; room: This 
condition contributed to difficulties in recruiting and reUining highly 
trained; professional staff. This program, which was supported with 
funds>Trom the Government and private foundations, was surted to 
provide a special child health care service at a cost that Barbados 
could afford and to monitor that service by the use of certain 
nutrition indicators. Another intention of the program was to provide 
useful information from which Government, planners could decide' 
whether this program should eventually be incorporated into the 
island's health services. ^ 

Five years later, in 1972, as an outcome of the above pro'gram, the 
Nutrition Centre of Barbados was established by" the Government 
within the QEH compound. One of the centerVten basic functions, 
performed in collaboration ^with*the staff of the children's ward, is 
the- management of all malnourished thildren before and during 
admission, as well as after discharge from the hospital 

Area of Child Health Care 

The service offered by this program has been mainly in the area of 
nutrition mtervention. However, from* the start, the program was 
integrated into the health services by the author, who holds joint 
appointments in the departments of pediatrics and social and preven- 
tive medicine. 

^ In thivhospital, treatment of malnourishVd children is conducted 
by a highly trained team of consultant pediatriciarfs, junior supportiye 
staff, and nurtes. They, in turn, are supported by a home visiting 
program run byNiommunity health nurses. The latter program is 
designed to identify contributing factors and to institute innovative 
measures in the management of undernourished children before 
admission to or after di.scharge from the l^pspital. 

In the nutrition clinic, the child is measured and his anthropomet- - 
ric measurements are record ed:,;;F^e mother /s given nutrition 

, education by group discussion-demonstration and later by one-to-one 
teaching. At that time breast feeding and nutritionally relevant 
weaning techniques are emphasized. 

These procedures are followed by a clinical examination, counsel- 
ing, and treatment. Full immunization for ^school entry is performed. ' 
When necessary a subsidized food supplement and/or medicine are 
provided. FuU laboratory investigations are possible through the 
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hospital laboratory service. Nutrition surveillance and rehabilitation 
services. are available on an ongoing, basis for these children from 
birth to age 1?. 

Althou^ the program attempts to be comprehensive in aim, it is 
focused on children at high risk for malnutrition. Efforts have been 
made to integrate this endeavor with the existing services for 
maternal and child health, and family planning and health (including 
dental and mental health) education. 

Teaching the family how to copje with problems is very rmich a 
part of this program. It is p^t of the community health nurse's jiibj. 
to try to find employment for the mother and to teach her how to 
budget wisely. . 

Study Design of Nutrition 'Intervention Program 

A nurse interviews the mother or guardian of an infant with a case 
of EPM. Using a structurc^d questionnaire (appendix 1), she elic-its<^ 
information pertaining to the infant*s nutritional backgraund|Q An- 
thropometric measurements,' clini(:al examinations, and laboratory 
tests are performed. All weights are charted according to the Gomez. 
Scal^' (appendix 2), and the road-to-health .card (appendix 3) is 
retained by the mother. \ 

. Home visits are made by nurses to gather data (^h environmental 
conditions, record anthroptometric measuremerits, treat minor ail- 
ments, and provide*^ health education: In the nutrition clinic, the_^ 
nurses hold cbnsultations with the mothers and children, record 
anthropometric measurements, exatiiine and treat children with 
prepackaged' medicines under th^ physician's supervision and dis- 
pense powdered whole milk at a price subsidized according to the 
child's nee4. Immunizations, demonstrations, and health ediicStion at 
group and individual levels are also performed. - 

A family planning service is provided in the nutrition dinic by ^ 
representative of the Fathily JPlanrring Association, as well as by our 
trained nurses. This active association has 1 1 clinics throughout the 
island. In the family plarining clinics, after conducting an interview 
arid examination, the staff dispenseis various contraceptive devices for 
a nominal charge, or free for indigent clients. 

At the end of the clinic session, a record is made of all mothers 
who failed to attend. A nurseji^sits each home on the following day 
to ascertain the reason for the absence. There she provides the same 
service the mother and child would have received if they had 
attended the clinic. A new appointment is made, and the mother is 
urged to keep all future appointments. The nurses haVe access to 
transportation, and a minibus is available to those mothers and 
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children who live several miles from the hospital or who do not have 
bus fare. "^"^ . 

When the child is ready for discharge from the hospital, piothers 
are instructed to bring the child regularly to the nutrition clinic or to 
return to the 'ward if the'ctiild becomes acutelytJIl, In addition to the 
main Thursday clinic of about 50 children, there- arc miniclinics 
comprising about 8 children on other days except Sunday, when ce 
children visit the QEH emergency room. In the event of unsatisfac- 
tory clinical process at home» the nurse takes the child to the .ward 
for treatments^or admission, as^ihdicated. Thus the community nurses 
work in cbse collaboratiort with the ward nurses in the management 

of EPM infants. ^ ' 

...... ^ 

In this nutrition intervention program, the community nurse 
frequently has three categories of mothers and cliildren— those whom 
she has to see daily during an acute illness, those vyho attend the 
nutrition cliqic. regularly for general health suj>ervision, and those 
who have failed, or are unable, to attend the clinic and have to "be 
followed at home temporarily.' Consequently, home visiting otcurs on 
a selective basis. 

In her role as homeirnaker, the nurse gives advice which is adapted 
to the realities of the situation in the home. She also prd^^!^e|^rsing 
care and medical treatment whpn prescribed by the physioah. Great 
emphasis is placed on home visiting, and in 10 years'the nurses have 
failed to trace only 14 children, 6 of whom have emigrated. In this 
way ^ood rapport is established • between the mothers and the 
com nriunity nurses. ' 

Cases witlj difficult socioeconomic problems are discussed at staff 
mieetings, and every effort is made to develop a team approach to^the 
solution of these, problems. The data on which the results are based 
have been takert^from the program records, which are compiled 
.separately from the^ hospital records. Simple monthly reports showing 
relevartt data and trends are compiled by the nurse supervisor 
(appendix 4). . * ' 

The following indicators were considered most suitable for the 
measurement of changes in the nutritional status of the Barbadian 
preschool child: ^ 

—Mortality rife in children of less than age 1; 
— Mortality rate in children ages^ l to 4; 
^ — Mortality rate in children ages 1 to 2; \ • 
—The proportion of all death^s in children undfer age 5; 
>r-Specific mortality rate for, malnutrition; 
— Specific mortality rate for gastroenteritis; 
— Weight and height in relation to age; 
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-Clinical indicators .of nutritional status; ; ' \ t 

— Classification of children according to the Extent of malnutrition 

(Gomez scale and WHO classification); * , 
—Percentage of readmissions; 

— Number of days spent in hospital by children recuperating from 

malnutrition (patient-days); 
— Number of admissions of malnourished children to hospital; 
— Cost of rehabilitating malnourished children in hospital; 
— Cost of rehabilitating malnourished children in the community. 

Results ' . V 

Table 1 compares statistics of children admitted to QEH with EPM 
from 1967 through 1976. This taple shows a general trend of 
improvement during this period in tit e ind icators of admissions and 
readmissions, especially when these arecornpami with the base year 
of 1966 (the year before the program started). ^^Eor that year, 
admissions , were 1^0 patients and readmissions 25. pfercent. This 
trend is more marked in the number of patient-davy and in the 
hospital case-fatality rate, which show. ^a decline froni 7,825 (1967> 
patient-days to 1,879 (1973) and from \3\^ perdent ftsii.9 percent 
respectively.* In 1974 and 1975 all thfese nutrition indicat^ showed 
deterioration, but in 1976 i remarkable recovery was posted] with the 
lowest figures on record for admissid.ns, readmissions, patlpnt-days, , 
and hospital case-fatalky rate. 

Two national nutrition surveys have been carried outin^arbados. 
Both were random stratified samples of the entire island population, 
but the 1969 survey comprised all age groups, whil^ the 1975 survey 
was restricted to the mdst vulnerable group, from birth to age 5. The 
results are illustrated in table 2^ with a comparison by age pf the 
Barbadian preschool children in the community who fall into differ- 
ent levels around the Harvard starufard weights for age. 

This table shows that the rate ^r tfiose children who failed, to 
thrive (below 80 percent of standard or .the third percentile) declined 
frorti inv5 .percent in 1969 to 10.2 percent in 197-5. The Gomez scale 
for moderate and severe malnutrition (below 75 percent pf standard) 
shows a surprisingly low figure of 3.5. percent in 1975, 

The infant mortality rate declined from 54.1 (1967) to 29.1 per 
1,000 Jive births (1975). The mortality, rat^^ in children ages 1 to 4, 



* The apparent «ris« in patient d^ys of 3,463 (1971 J-D), as compared wi^h 2,860 
(1970 A-D) can -be explained onjlhe^basts that the 1971 period is for 12 months and 
the; 1970 period, for 9 months. The jise in caie fatality rate for 1970, was associated 
with a temporary. staff shortage. > 
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ages I to 2, aind^lhe prpportions of all deaths in children under age 5 
from 1967 to IQ^^ Kkve not b^en analyzed. However, research 
indicates that, there ar<j fewer severe grades of .malnutrition on 
admission to the war(i atnicl ^hat itiothers are presenting earlier at the 
' hospital with relatively mqrc Grade I EP\f thildren. 

The cuniulative community case-fatalit)»c rate of 5 i>cr 1,000 chil- 
.dren ages 1 to 4 years recuperating from malnutrition (1969) 
><ompared favorably with the mortality in the general population over 
the sanlc age group which was 2,2 per 1,000 children (1970) 
Barbados and 4.5 per 1,000 (1968) in Jamaica, This rate is well bclpw 
the published figures for community studies, which range from 15 
percent to 30 percent. - ^ 

Utilization of Resul||. /-\ f / 

Priorities. An analysis of fapmissidfns of children to the ward in 19^6 
demor)strated a pattern of oi&^as^ that made us decide to concentrate 
on the treatmenjt of malnutrition and gastroenteritis. Since then, 
frequent analyses of our results have assisted us in developing, 
priorities. t 

Staff. The presentation of interim reports at the outset pf the 
^ogram enabled the direct^ to recruit staff, first ^/rom the 'Peace 
Corjas, and finally from the Government. ' 

information System. Monthly and annual repKjrts have been submitted 
to the Ministry of Health, and politicians and administrators have 
released edited versions of data periodically through the communica- 
tion media to inform the public concerning improvements in the 
^status of child health in Barbadcfs^ 

Columnists have been s|ipplieai with technical information arising 
from this program that has'^rtabled them to prepare nirj^ition articles 
for the women's page of the daily newspaper. Occasionally, feature 
articl^ have been written about the work of the nutrition intervention 
program as a result pf interviews with its director oi" staff members. 

Roa^ to- Health Charts. A modified version of the original Morley 
Road-to-Health Chart ^'has been used successfully in this prograru. 
The demonstrative succiess of this chart has convirtced the Ministry of 
Health to adopt it. Its use is now,.manc|ator^yn all Government child 
health clinics, and the chart is available (t^ to all voluntary^ child 
health clinics as well as to generaL practitioners. This chart ms. also 
used ,to demonstrate to the pnncipa4 that immunization for school 
entry has been 'fully completed. • \ ** ^ \ 

Motivation. Various nutrition indicarors have shown a steady decline' 
in malnutrition over the past '10 yi?ars. This visuaMemonstration of 
improvement in the. health status of children, which we had felt^ 
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intuitively, has helped to promote and maintain a high sense of « 
motivation amojng the staff. ' > - 

Surveys. The QEH nutrition-indicator statistics suggested a marked 
^ decline in the prevalence and severity of EPM iq the early years. 
When the improvement was presented at various scieiitific meetings., 
a question was raised as to whether these figures reflected accurately *^ . 
the situation in the. coqimunity. These discussions tendd^l to expedite 
the community nutrition surveys that were carried out in 1969 and' 
1975.. ' ^ • ' 

Establishment of the Nutrition Centre Jn Barbados. As the nutrition 
intervention program progressed, the work Increased ir\ volume and 
. ' complexity. With a stjfadily'growing staff. the Vmall space allocated in 
^ ti corner of the ward and in the general children's outpatient clinic 

1^ soon became inadequate. These factors, coupled with the results of 
the interim reports and funds raised by voluntary efforts, local 
Organizations, and overseas foundations, led the Government to^ 
establish the Nutrition Centrjp by providing the land jnd a matching 
grant. ' ^ - ' ^ 

Publications. Data gathered in the program have been of use in the 
production of pam'phlets* brochures, manuals, slide preseiitations. 
learning packages, and calendars.. This educational material has been 
disttibuted to comrfiunity nutrition* groups, schools, teacher training 
colleges, and interested lay and professional persons. 

Training. Information from this program has been very valuable in 
designing demonstrations for mothers of malnourished children ^as 
well as in influencing the content of seminars for nursesvxomm unity 
groups, health aides, and medical students. ' 

St. Vincent X^ommunity Health potion Programme. Experience gained in 
this nutrition intervention {>rogram in' Barbados has Jed to the 
implernentation of conimunity health aotion pipgram in two, 
'^communities on the neighboring island of St. Vincent. AMr ^-years \ 
. it h hoped that, with the Barbadian example, the government of St. 
Vincent will find it financially feasible to assume responsibility for 
this program. "'^ ^ ' ' * 

Estimates of Expenditure. In the draft estimates for the nutrition 
center, the results of the nutrition intervent'ipn programs<have been 
used in relevant budgetary notes in order t») justify requests for 
funds. This application ha^ enabled us at Estimates Committee 
meetings to base our requests on fact^ual information rather ^an on ' 
hjimanitarian grounds alone. , " ' ^ ^ ' 

Research. The need for further re.search mto selecte,d areas of the, 
progr.am has been highlighted by the results, and action has been 
taken where appropriate. " ' 
^Invitations io Meetirigs. The progra^m and Results have letl to 
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invi<ations to attend various international meetings. In turn, discus-'' 
sions at these meetings havip provided valuable information for 
im|hx>ving the program, and contacts with professional colleagues 
have also led to direct inputs in some, cases. ^ 



Discussion ^ 

The original ;|>urpose of this progrs^ was to apply effectively 
existing 'liinow ledge in reducing the overcrowding and cross-infection 
Oathe QEH children's ward and not to do basic research. 

The origi^al,objectives were clear. In each of the next 5 years, the 
goals were J^iv 



-~Red\Jte"w readmission t^tc for malnutrition by 10 percent; 

— Red0e{ the case-fatality rate for malnutrition, by 4 0 percent; 

— Establish and compare tfic morulity rate of children ages 1 to 4 
years nfihtuperating from n)alnUtrition with the ratfc of children in 
the same age group for the general population ojf Barbados and ? 
other Caribbean territories. ; ' : 

Since the program 0^iis based in the'^only Government-spon^red, 
^cute general hospital serving the entireV^and, the sample size was 
adequate. However, the treatment of malnourished childl-en is widely 
known and accepted, and ISccausc treatment could^noll be withheld 
on humanitarian or political grounds, the use or contro\^groups was 
not considered feasible in the study desigh. 

The prograih had satisfactory means for carrying out meast 
menu and recording^Hata, but -not the processing of the data. Thi 
mechanisms for checking the validity of the procedures were supet-^ 
vised internally by senior staff members of the program as well as by 
visiting colleagues. However, apart from the internal evaluation by 
our staff, no special staff members with skills for objective evaluation . 
have been av^U^bhe; to us. * ^-^ 

0\xr ^ijiisilin^^ have provided with valuable infoi^tion 

which ha» Het^^ied^ our performance, l^^evertheless, itNhas 

not been 4x>ssi(>fe to consider and give detailed costs of-a rang^ of 
alternative^More importantly, a detailed cost-hSnefit or ciQ$t^ffc^tiv 
*bnalysis, iiflprms of input costs and direct-or indirect output costs, 
has not bctfrcarf i^ed out. ^ 

Althou^feausality cannot be proved, the target groups of families 
of malnbCnhed children, politicians, administrators, and donor 
agencies ^Em* reached, and objectives were met. r 
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Perhaps of great significance is the tact that apart from the 1974- 
1975 period immediately following the energy crisis, there halbeen a 
marked decline in the Jiutritton indicators of^this socioeconomic 
diseas<|, in spite of rising economic costs. ' . 
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APPENDIX 1 



MALNUTRlVlON PROFORMA 



NAME OF MOTHER: 
DETAILED ADDRESS?' 
OCCUPATION OF MOTHER: 
AGE OFlitoTHfR ON LEAVING 

SCHOOL: ► 
AGE OF MOTHER AT BIRTH OF 

CHILD: 

AGE OF FATHEir>AT BIRTH OF 

CHILD: 
NEAREST TELEPHONE 1 



IRTH< 



NAME OF CHILD:- 
DATE OF BIRTH: 
HOSP. NO.:. 
SEX: 

FATHER OF CHILD: 
ADDRESS: 



XGE: 



ADMITJJO ON: 

IMMUNISATIONS 

I 2 ,3* 
DPT. , 
POLIO 

SMALLPOX ^ 
DIETARY HISTORY 
F.B.F. P.B.F. ^ 

BREAST FEEDING: 
WHY DID YOU STOP? 
BOTTLE FED: 



DISCHARGED ON: DAYS 1 

DIEDON: P,M, YES 

MILESTONES 
HELD UP HEAD: FIRST STE 

SAT UP UN^dPPORTED: FIRS/^WORL 
^TOOD UP ALONE: WALKED AL 



B.F.N. 



WEANING 
SENT AWAY 



WHV? 

YES'''^'.-;- 
NO* . . 
PERSON 

DIFFERENT HOME 



N.B.F. 

< 

COMPOSITION 
ABRUPT 




SUPPLY <p 
FREQUEI 




'v V 
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; ECCS 

• « » ' 

/. > - ^ MEAT 



iQUENCY/ 
WEEK 



? FRUITS'.. 



VECEiABlls 



• STS\RCkVFq(ops V; - . 

■'&ji{o, XrrowvooC . , V 
Cm^jltnealt, Linseed {A 

' it' ■ ^ 4 

^ RUSk.TEAS. . ' 
jsfodf^fof prepai^uoB 

4 ' i^' 



(V9 Mthi 


$-6 Mthi 


6-18 Mths 


1-2 Yra 


» 

































































>tY*t or M0SH TEA ANbfKEQUllNfcY . 

o^LKLGkRE ANb CUSTOMS: — > : ; ' 

;IS'|t H»kMFL(LXO BREAST F££D AFTER* ... 
• /{ I) ^WorJiing in the syn '.' ■ '. r 

' (2) Cohabiiing;' : . : rv . N ; v 

; 'X3)^ Other reaaqm /. 
' (4)' ^'milk too' heaV^^^^^ I ^. 

/ (S)yAre soAie ]'ceas'^ ''cooling" an<;i others '•heating*.'? 
: . If >es,** m9ke,Ns^ of cooling an4 V^u^^ 



Do^pme food9 give worms^ ; ' , 
> • ; If '^es/' which ones? 




"PREVIOUS HISTORY: 

* BORN AT: 

TREATMENT 
ATTENDED BY 



PRECNANCVi 
CESTATiOSf 
SINGLE; 
CRIED 

IMMEDIATELY 



V.D.R.L. 

delivery 
'multiple 
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V MtAiiLES r^^^ V J|Pw BRONCHOPNEUMONIA 

; CA^TRpft^?1t;r^^^ ^ OtH^R '^'W 

^4o;ar^ltbKN. 1^ 

- S^j||. FtNAN<:^^ YES NO MARRIED UNMARRIED 

^ , FAt^HfJi:jX?(SJ^ EXPENDITURE . 

HOitfSE^MNT? PAIP ^ • FREE EXPEfJDITURE/WEEK 

MOt'rttR'4*0C:^P'ATlON HOW MUCH ON FOOD \ 

MOTlm|,'^ 0R^S: ; . |p£LL PLAIN * 

M0THOTjS^*Nl|i^^^ SATISFAC- UNiSATISFACTQIWr 

-• '^1 V;^.- ■ ■ m ' ; . TORY \ ■ , . ' ■■^/r 

«AGlJ|<5TF/BAEW^HfcN MOTHER RETURNED TO WORK: / 



^ : -IjAS CH^kB SP^T ANY TIME IN ALMSHOUSE? 

v V;. ./REtiiJi AT CLINIC WITH INFANT: CP. QEH NIL 

; f. /i,;/ EDOffi\Tiofi: ^/v PRIMARY Cla5« 

• V SECONDARY Form . 

\ \ . FAMII.Y'^frLANWING: 

^ ^ DID YOU PLi\N TO HAVE THIS BABY? YES NO 

A tlAVE^U^TTENDED A FAMILY PLANNINC CLINIC? YES NQ 
ftfiFEIHBD? YES. NO ' > 
-* ;j^WEREY©tJ USING ANYTHING TO PROTECT YOU FROM BECOMING PREG- 
^■.■■V', ^>rV'Al^t? V YES NO. ' 

"34v>APOE5>-:FHE FATHER ACKNOWLEDGE THIS BABY AS HIS OWN? YES NO ^ 



■•'^ ^./"^ H/ 



... 
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cuniCal examination* 

. T. - 
W.dA. 
HT.O.A. 
HC.O, A. 

SWELLING 
WASTED MUSCl^ES 
SUBCUTANEOUS FAT 
MISERY 

HAIR CHANGES: 



SKIN COLOUR: 



^ LOOSE STOOLS: 
•COLD" 

RASH FLAKY PAINT 

• OTHER 
ULCERS AND CRACKS 
SORE ANGLES OF MOUTH 
BRIGHT RED TONGUE 
LARGE LIVER 

INVESTIGATIONS: . 
Hb. 

V.DR.L. 

MANTOUX 

STOOLS 

URINE 

EYES 

TEETH * 



P. R. 

CC.O.A. W.O.D. 

CC.O.D. ' HT.O.D. 

ARMC.O.A. HC.O.D. 
A.C.O.D. 

YES NO 

YES NO 

YES NO 

YES NO 

BROWN REDDISH 

^REV STRAIGHTNESS 

SPARSENESS SIl.KINESS 4 
EASILY PULLED OUT 
LIGHT REDDISH-BROWN 
OTHER ' 
NORMAL 

VES NO 

Ves no 

YES ' Ni) 

YES NO 

YES NO 

YES NO 

YES • NO 

^ES , NO 



* 



Ht. 



MORPHOLOGY 



cdcba abcde 



' W. Weight: HT. Height; Hp. Head circumference: CC. CheA circmnference: 
Arm circumference: O.A.. On admission: OiD.. On discharge 



AC. 



4 
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FIELD NURSE^S FORM " . 



NAME OF MOTHER: 
DETAILED ADDRESS: 



HOME ENVIRONMENT 

R. - RESPONSIBLE Fdl CHILD 



NAME OF CHILD: 
DATE OF BIRTH: 
AGE: 
SEX: 

SURVEY NO.: 



, HOS. NO/. 



D C,,- DAY CARE OF CHILD 

t : « 



0-6 MTHS 3-6 MTHS 6-12 MTHS 1-2 YEARS 



GRANDMOTHER: 

MOTHER: 

FATHER: 

OTHER RELATIVE: 
PAfp PERSON: 
CHIJ-D UNDER 14: 
OTHER (Specify) 



PARTICULARS OF HpU^iEHOLD: 
NO. OF ADULTS: 
NO. OF CHILDREN: ;j 
NO. or ROOMS: V 
SEPARATE SLEEPING AREA jRQR INFANT: 
RUNNING WATER: YES > *NO 

TOILET: PIT BUCKET 

KITCHEN FACILITIES: " KEROSENE 
MATERNAL CAPACITY: 

MOTHER CRAFT: «^ SATISFACTORY 

HEALTH EDUCATION: 
PROTECTION OF'MILK AND FOOD 
PRESENCE OF FLIES RATS 
IS BOTTLE STERILISED BEFORE EACH FEED? 
TYPE OF BOTTLE USED: 



W.C. 

' WOOD 



UNSATISFACTORY 



ROACHES 
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FIELD NURSE'S CHART 



1 DATE 


AGE 


WT. 


HT. 


HC. 


cc. 


AC. 


rencarks 






































* 






















































• 




































— 




I 












T — 
































































« . ■• 



















































































































Is 



Nutrition Intervention Program-^BartNidos 187 
APPENDIX 2 V 

MRBAD08 

Q. E. H.^liyTl^mON CUNIC 




MONTHS 




. BARBADOS 
MINISTRY OF HEALTH 

HEAU^ RECORD 



mi 




HI 


i^ODREU 





IMUMIM. M,l> 


OMIITH 


1 III 1 III viHim tit 1 II '1 1 1 1 M 1 1 1 

piHtiim 


1 

r-' 

1 


MTMirt 

ml 


'. , in % 





MAlLirOX IMMUNIIATION 

' - 
.(wrtAitiiicinion 

' nifOWloi] OATl ,, 

G m 

^QOAU 

wcceHfuiDw! ■ 

MTWCCIUPUlD Wl 

' D Ml ■ . . 



DOCTU'l 
NUIIi'lilSNMUK 



pACCINATIONWAINJTKAUfOX 



RiailTIMDAT 



DbCTOR 



'■ 'I' " 

CHILO't 

NO, , 



aiNic. 



HOSPITAl 



HOTHII'l 

NO.,.,. 



IMMUNIZATION KHIOULC 



you SHOULD TAKE YOUR CHILD TO YOUR DOCTOIV 
aiNICASFOLLOn,- 


AGE «' 


..- KWUNIZATION , 


]|ITH) . 

" 

r" '. 

,4rilM«TNI 

ntt'iTNt 


111 DM ^NDOHJil ^OUQ. 

w , " , ' • 

* UilOOHTilOfT«*0, 

* ORAL ^ouo' 

ii IOOU(I(OI;T AND ' 

VNKCHriVI)' 
MiliAllP01(VACCINATK)H 







^ 

Oil* 


1 1 "1^ 
1 
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1 * 




Dll^ 


^THINIA, WHOOPINO COUQH, 
>ITANUJ, POtIO 










- — ^ 




JL- 




IHliP ■ i 








\ 




tireOOiTEfi 












' i 












4* u^m 
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. OTHCRIMMUNItATIONS OR TESTS 



TMi Sooi 11 0 ptfionwl M ^ 1^ 
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' < ^_ ^^^.^^ 
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^ i 
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APPENDIX 4 



BARBADOS NUTRITION PROJECT— MONTHLY REJpORT 



197^ 



Cumulative Total to End of 
M F Total Current Month 

This Year Last Year 



Admissions 



■ I 



MaAs 



Maratmk Kwashiorkor / 



Hi^washiorkor 



Grade I 



Grade 1 1 



Grade III 



0-9 Months 



3*6 Months 



6-12 Months- 



1-2 Years 



2-5 Years 



Ward Deaths 



Anaemia 



Readmissions: PCM 




admissions: Other 



Hospital Days 



Occupancy % 



Cost 
— ^ — 




19- 
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Firji FirJl Visii Rcvisiis Chil- Attend- Children Attend- 

Vuit This Year dren ances ances 

Ever 



. immunisa* 
tions 



Triple 8c 
Polio— 1st 

Triple 8c 
Poli«~2nd 



4- 



Triple 8c 
Polio— -3rd 



Triple 8c 
Polio— » 
Booster- 



Small Pdx 



lClilea((^ 



Measles . 








/ 




Home , 
Visits 


^ 


• 









Signed- 



Remarks 
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DISCUSSION * 



H "^'^ Milton i Roemer atj^ Gabrjel Velazquez 

The numerous improvements in the nutritional status of children 
\\\ the islander Barb^jdos have been clearly demonstrated by Dr. 
Ramsey's report. The evidence is strong, not only from the reduced 
admissions of malnourished children to the island's single large 
hospital, but also from a decline in the case-fatality rate of those 
children admitted. Probably more important epidemiologically has 
/been the reduction in the'nuniber of underweight child ren^^fojiwd^ 
•p^)pulation surveys. Nevertheless, how much of this improvement can 
be attributed to the energetic| nutritional program developed and 
/ described by Dr. Ramsey is not clear. 

In the Barbados" setting, a "controP population was not feasible, 
nc^r was any other Caribbean isj^nd suitable as a control. No data 
were presented on the trend in per capita income or the standard of 
living over the 8-year observation period. Price levels did indeed rise, 

• but we do not know whether changes in earnings were greater or les? 
than the inflation rate. The birthrate in the Barbados popplati 

^ declined from 35.4 per 1000 per year in 1955 to 21.9 in 1975; th"i 
^ t:hangc alone could be expected to improve the nutritional status o 
the few^r children in a family, even if family real in^come remaind 
unchanged. (The family-planning program, of course, was part of; 
the total effort, but it is not the same as nutritional intervention.]/ 
Also, over the same 8-yeai;^eriod, the Government promoted a 
program of diversified agriculture, in contrast to the island^s previous 
. exclusive dependence on sugar production. Tourism also brought to 
the island more money, some of which doubtless filtered down to ^he 
y^general popi^lation. 
J These weaknesses in the evaluative research design do not di>{^tfl£e^ 
^ the value of the nutritional interventions. They must make one very * 

* cautious, hov^ver, (n attributing to those interventions the full credit 
for the improvemeru§=in the nutritional status of Barbadbs children. 
AJfter^all, Dr. Ramsey's creative work \f^as undertaken in response to 
a serious problem, "Ot^s a scientific experiment with evaluative 
strategies bTRTt m at th<wu\set. 

\j\ spite of these evaluaftive limit^^tions, the Barbados experience 
has (^lessons to teach other developing countries in demonstrating 
indic^ors of nutritional change by way of both hospital-patient data 
and population surveys over time. The task in future research Of this 
tyfxr will be to establish "control" or comparison pojaulations, Avhich 
will permit ruling out (or holding constant) the intervening variables, 
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so that one can draw conclusions on the actual cont 
nutritional intervention orogram to the chahg^es fo 
the nutritional stiaitus ofctnldren. 



\ 






13 



RESPONSE OF INDICATdRS OR NUTRITIONAL 
STATUS TO NUTRfiiONAL INTERVENTIONS IN 
\. . POPULATIONS AND INDIVIDUALS 

• C Yarifrough, J-P. Habicht, R. E. Klein, R. Martorell, 
-A, Lechtig, G. Guzman 

SUMMARY ^ 

An intervention study designed to change ^l^e nutritional status of 
^ moderately malnqm^ished preschool ^Hildreri illustrates the use of 
anthropometry as an indicator of nutritionarl status and sti^ovi^ that 
demonstrating the physiological efficacy of an intervention does n<k 
guarantee a significant total impact ^f^that intervention on the 
population.* Caremil design makes evaluation of some .interventions 
possible; but theory^and knowledge iA human public health nutrition 
an<| its relation to growth in children now limits the xlesign and 
especially the evalualidh of many nutrition interventions. This paper 
contributes to recent^knowledge in this area by estimating the actual 
^nergy conyersiorv/rates fer^growth in moderately proteinncalorie 

Jalnourished children aged 1 through 3 >(wr^ under fidd c^ndlitlons. 
hese conversion rates are even lower tTRn those which would be 
estimated from normative data in the literature. ' 
w ^. ^ 
PURPOSE OF TH£ STUDY \ 



^Th^ present study was an intervention .designed to change the 
nutritioiial status of prt-eschool children in order to test experimentally 
the bypothesjs that modCl'ate prdt^n-calorie malnutrition is related 
to impaired rrontalNdevelopmeot (Canosa, Salomon, and Klein 1972; 
Kleifti Habicht, and Varbrough 1975, p. ^l). Thus the principal aim 
of the project was ncf public health intervention, but gather scientific 

•The prcfent study has been funded for approxiipa^ely 9 years by the National^ 
Institute of Child Health and Human'Dcvelopment in/^.serics of contracts (currently 
^01-HD-5-064(jyv to the Institute of Nutrition of Central America and Panama 
/(INCAP)^ a'resrtirch center of the Pan American Health OrganizatioQ. 



/ 

19H .KVALIJA HON OF C^ULD HFAU H SKRVICKS 

^investigation of the causal effect of nutrition on certain variables. 
Whether or not the purpose of the study is'scientific demonstration 
OF public health" intervention, evaluation of such a program requires 
a comparison- betWen two. groups, .One group must h^v^ benefited 
froqa nutrition not receive^ by the other group. Here we present 
appropriate comparison groups for scientific demonstration of a 
nutritional effect,'^e will show that such an effect is essential for 
evaluating public health intervention programs in ^tKition, even 
th9ugh an intfVvention program that scientificall^L demonstrates the 

-effect of nutriti^ does not oecessanily produce a meaningful effect 
on nutrition of tn)e population^ as a whole 

The proposed l^y^^ the basic hypothesis consisted of improving 
the nutritional ^itus/ oC chh^en* in order/o look for associated 
changesin inieMal ■ development measures. Thus, the first step in 

- ^"^Jyz>/^^h^THdy islio show that suchxnutritional improvement did 
m Wet occuf,,.which in turn requires "the measurement qf some 
outcome uftiversally (reccignized as being related to nutrition in 
malnourished populafioTTis, 

• AREA OF CHILD HEALTH C/^ 

Failute to achieve maximal mean growth rates during infancy and 
early childhood* is probably the mdst frequently used indicator of 
poor nutritional health status for populations: The use of growth 
rates to a^iess gommunities is widely accepted (Jelliffe 1966) and rest^ 
on a solid scientific base: Early growth rates show no apparent 'ethnic 
differences and are affjefted by low dietary intakes and high morbid- 
ity rates (Habicht, Martor^ll, Yarbrough\et al, 1974; Marsden 1964; 
Martorell. Habicht, Yarbrough, et al. 19X5; Martorell, Yarbrough! 
Lechtig, et al. 1975KThus.we shall be cons^ering here whether the 
intervention described b^low had an Impact on the growth of young 
(?hildren between the ages of 1 and 3 years. Other nutritional effects 
on children's performance, healtji, and survival are not dealt with in 
this/paper. ^ . 

^ The issues to be addfessed ar^ (1) Does this study pre>ient 
convinci^jg evidence that improved nutrition improVes growth in 
n)alnourished. children under field conditions, where other growth- 
retarding infiuences continue t<> be prevalent? Previous studies on 
thi^s issue have been contradictory (Majtorell, Yarbrough, Lechtig, et 
al. 1976). (2) Does this study_p£e^en/ convincing evidence that the 
nutritional effect on growth can be physiologically substantial under 
field conditions? Previoiis studies, which* claim an effect on growth, 
have been contradictory as to whether or not the effect was substantial 
(Habicht and Butz in press). (3) Does the information clerived from 
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this study about the effect of nutrition on growth explain the 
contradictory results of previous studies and suggest strategies for 
future piublic health intervention programs? 

STUDY DESIGN 

Target Population * - ' ^ 

The study we are reporting was conducted in four rural Ladino^ 
villages in eastern Guatemala-. Since the inception of the research, 
free outpatient preventive and curative medical services ha\e been 
made available to everyone in the villages (Habicht 1973, p. 24), The 
economy is subsistence agriculture; the local diet is based on corn 
and beans. Our principal concern is with the young child. In these 
villages there is considerable stunting in growth of these children, 
consistent with a diet in which it is estimated that the protein-calorie 
ratio is well above nccommended limits (World Health Organization 
1973, National Academy of Sciences 1974). but the energy level is at 
least 20 percent below the apparent requirements for children aged 1 
to 3 -years. Table 1 compares values for the four villages at the 
beginning of tht study with values '.typical of developed countries. 
Note the absence of sex differences in incremental growth (Yar- 
brough, Habicht, Malina. et al. 1975) or diet. Although table 1 does 
not reflect the small dietary contribution from breast milk, we believe 
that the average daily intake, including breast milk, does not exceed 
80 kcaiykg/day between ages 1 and 3. 

• ■ ■ . , ' oi' 

Design, Methods, and Data 

The ^starch dc;sign presented here is a comparison of the growth 

• of children who voluntarily consumed different amounts of supple- 
mentation food. 

In two of th^*1bur villages, "atoleVa protein-calorie preparation o¥ 
Incaparina, dry skim milk, and sugar, is' distributed. In the other two 

* villages, ^'fresco," a taloric preparation containing sugar, flavoring, 
vitamins, fluoride, and iron, is given. Table 2. shows the nutrient 
content of both preparations. Both provide the same vitamins, 
fluoride, and iron, but atole contains proteii^ and more calories. The 
supplementation, whethA atole or fresco, is available^ twice daily in 
the local field station of each village, and attendance and'consumption , 
by all villagers are encojuraged, but are entirely vofeht^ry. A daily 



*Of Hispanic cuUurc>in contrast to African Indian cjj 
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Tabic 1. Growth and Diet of Study Population fin Comparison with Values 
from developed O:\untries , \ 



Values Typical of 
Study Population Developed Countries 





Males 


Females 


Males 




Females 


/ 

Height (cm) 












12 mo. 


68.9 


67.3 


74. P-^ 




72.3 


36 mo. 


, 84.4 


84.2 


95.2 




93.8 


Net growth 


: 


16.9 


21,1 


C 


• 21.5 


Weight (kg) 








12 mo. 


8.0 


7.4 


10.0 


9.4 


36. mo. 


11.7 


11.4 


14.6 




13.9 


Net growth 


3.7 


^ . 4.0 


4.6 . 




4.5. 


Dietary intake (kcal/kg/day) 








15 mo. 


66'-^ 










18 mo. 

21 mo. ; 


73 
• 78 










24^^ mo. 


80 - 


- , ' 78 » 


10a:i03» 




99-106 


30 mo. 


' 80 


7S 








36'^mo. 


76 


75 









* GroWlh data from baseline studies prior to intervention. I ' 

* Data from McCammon. 1970. ' ' >f ^ 

12 month height adjusted by 1.7 cm to correct'for differences in measuring techniques. 
*^etary intake data taken during tim^ period of intervention. 

•Values apply to ages I to 3*!knd are from Table 7 p. 34 Energy and Protein Requirements. 



. record of: attendance and intake is kept for e^ch individual. Because 
of the different relative ft>ncemratidns of calories ^s related to the 
voluQie ia the two supplements, and because of differiog ingestions 
of supplement for similar attendance to the supplementation centers, 
these data can be analyzed separately for the relative contributions to 
growth of energy 'intake, attendance, .and volume of supplement 
ingested. ^ ^ 

Beginning in February 1969, data were collected on all children 
under 7 years of age, and all children born to the village populations 
before March 1973 were added to the sample. Data collected on 
chihdren until thpir seventh birthdays included anthropometry, dental- 
eruption and skeletal maturity observations, dietary and morbidity 
'surveys, socipcultural inventories, and a large battery of infant and 
preschool mftital tests. Other dala, not presented here, were collected 
on pregnanrand lactating mott^rs. Here we are discussing only 
growth in .height and .weight of the 456 children for whom data were 
collected at both 12 and 36 months of age. \, 

Height was determined by measuring supine body length on a 
standard measuring table, and weight by a beam balance, both 
measurements taken by standard pr^edures (Martorejl, Habicht, 
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Table 2/ Nutrient Content Pef Cup* of Supplementary Feeding ( 180 ml) 




s 



Atole 



Fresco 



Total calorics (kcal) 
Protein (g) 
Fau (g) 

Carbohydrates (g) 
Ascorbic acid (mg) 
Calcium (g) 
Phosphorus (g) 
Thiamine (mg) 
Riboflavin (mg) 
Niacin (mg) 
Vitamin A (mg) 
Iron (mg) 
Fluoride (mg) 



163.0 
11.0 
0.7 
27.0 
4.0 
0.4 
0.3 
1.1 
1.5 
18.5 
1.2 
5.4 
0.2 



1.1 
i.5 
18.5 
12 



15.3 
4.0 



5.0 
0.2 



• Review date: October 11. 1973; figures rounded lo ihe nearest tenth. 



Yarbrough.^t al. 1975b). The observations were taken at the child's 
birthday ±7 days. ^ , 

Expected Impact of Nutrition on Growth 

Since there are no clinical or bibchemical signs of severe vitamin- 
mineral, deficiencies in the study population, we will immediately 
focus oh the issue of proteins versus calories. Studies of home dietary 
intake, biochemical indicators, and anthropometric indicators of fat 
and' muscle mass show a diet which* appears to be limiting in calories 
but not tn proteins (Martorell, Lechtig, Yarbrough, et al. in press; 
Martor^ll, Yarbrough, Lechtig , 1976). Moreover, there is reason to 
believe that in such a situation' caloric supplementation alone will 
shelter proteins and influence growth rates (Martorell, Lechtig, 
Yarbrough, et al., 1976); indeed analyses to date in this population* 
show that caloric supplementation is related to birth weight (Lechtig, 
Habicht, Dclgado, et al.' 1975) and placental weight (Lechtig, Yar- 
brough, Delgado, et al. 1975). Thus we shall d iscuss^the issue of 
growth as a response to ingested energy. This does not exclude a 
possible effect, of other nutrients on growth in this population. J^Jor 
docs it imply that the dietary problem of all populations is caloric. 
Many communities are in vastly different situations, and dietary 
interventions must be designed only after careful study of which 
nutrients are limiting. Indeed, interventions which are useful in one 
setting may be harmful in others. 

What in^pact will additional calories have on growth? Since a 
comprehensive discussion of all ages is beyond the scope of this 
paper; we will focus on ages 1 to 3 as the youngest i;>eriod with a span 
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s 

long enough toi fttetts ip hv apparent and in whirh energy 
consumption from breast feeding is not Important, Whal is the 
expected impact at ihis age of nutrition on growth under field 
conditions? 

We can make s'oiue estimates as follow^: Ajiproximately 50 kcal/kg/ 
day are necessary for basal nietabolism and maintenance. In this 
i^ange there is severe maliuiirition and cohiplete grcnvth failure. The 
average consumption in healthy populations of the developed nations, 
100 kcal/kg/day; leads to niaximal growth, I hus growth occurs in the 
range of intake from 50 to 100 kcal/kg/day. In the. four villages tlje 
typical diet of 80 kcal/kg/day is an int iease of SO kcal/kg/day above, 
the basic need, or 67 percent of the total possible-^crease in 
consumption. Thus approximately two-thirds of the part of the diet 
used for activity and growth accounts for 75-85 percent of the 
possiMe growth inci emeiu f rom I to years of age. 

In other wotds, the appi:oximalely 5.6 x 10* kilocalorie.*^ consumed 
by a village child in this 2-year period is 2,1 X 10* kilocxilories over 
l)a!>al iiietalH)lic iit'eds. This energy ingestion above the liisal meta- 
bolic re(|uirements leads to a growth of approximately 7,7 mm/lO* 
kcal in height and 182 gm/IO^ kcal in weight. In the'developed world 
the typical child consumes a total of 8,8 X 10* kilocalories in 2 
yeaKs — 4.4 x 10* kilocalories above ba,sal needs — and uses this extra 
energy to grow at the lower net rate of 4,8 mm/ 10^ kcal in height and 
104 gn^/IO^ kcal in weight,, Finally, the approximately 1,4 X 10* 
kilocalories additional needed to bring the diet of a villagf child to 
100 kcal/kg/day ciin give at most 4. ft cm of growth in height, i,e, a net 
rale of 0,3 njfiii/lO^ kciN, The comparable net rate for weight is^SO 
gm/IO^ kcal. This means that the greatest conversion rate of calories 
to growth that can be expected in the children is 25 times less 
efficient for height and 36 times less efficient for weight tKan was the 
conversion rat^ f rom calories to the childreiils growth observed at, the 
beginning of the study. 

Thus we would not expect extra food to be converted to extra 
growth at a rale greater thah the baseline .s^udy values; and we would 
expect that at some point the rate of growth per unit of extra food 
must drop dramatically in order to bring the overall growth rate into 
line with the lower value of energy-to-growth conversion* found in 
developed countries. ^ 

We conclude, therefore, that the conversion of energy io growth 
must follow a function similar to that depicted in figure 1,' which 
shows a curve of decreasing conversion of each unit of energy to 
growth with increasing total caloric consumption. The data for this 
curve do not, however,, exist in the literature, so that no exact 
numerical values can be given. Thus the predictions are at best only 
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J. 



^ Qj^^ty Energy Intake 
Beaal Metabolic Needa-J m Activity and Growth. 



Figure 1. Hypothetical rate of conversion of dietary energy to growth 

approximate. Furthermore, there is no infoi:mation about the effect 
^ of other nondietary, growth-stunting influences sudi as morbidity or 
endemic parasites on tins curve. 

RESULTS 

Table 3 gives the relationship between four categories of calorie 
ingestion* from rhe supplement by children aged 1 to 3 and their 
growth in height and weight. The increasing values of growth in 
height by category of supplementary energy intake are statistically 
significant (F « ^0.8, df * 3,452, P s 0.001) an^ biologically 
important (figure 2). . , 

The lower categories are, as we might expect, essentially at'tlf^ 
baseline levels given in table 1, while the top category shows a growth 
rate in which over half (he deficit has been eliminated. JFinally,' we 
note that prevailing levels of morbidity miike growth at the maximal 
rates unlikely (Martorell, Habicht, Yarbrough, et al. 1975a; Martorell, 
Yarbrough, Lechtigf, et al. 1975). Essentially the same situation- applies 
to^wth in weight (F = 1 8.7, df = 3.4||t f ^ 0.001) (figur<i 3). 
• Table 3 shows the overall rate of-«rtlversion of caloriesy^ growth 
by level of inUke. These rates wer^ obtained from multiple regression 
analyses^in which we calculated (the ^lartial slope of growth dn 
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Tabl^ 3. Relationship of Supplemental Caloric Intake i^id Growth from 
Age» 1 <o 3 • 







Level of Ingestion (kcal/day) 








11-50 


51-100 


101 
and up 


Total 


Sample lize \ 


66 


169' 


76 


145 


4^ 


Height increment (cm)* 


' 15.7 - 


16.5 


^ 17.2 


18.3 


17,1 , 


Weight increment (cm)* 


(±2.7), 


(±2.6) 


(±2.4) 


(±2.2) 


(±2,6) 


S.67 


3.85 


4.05 


4.50 


4.06 




^(±0.98) 


(±0.89J^ 


(±0.76) 


(±0.94) 


(±0.95) 


Supplemental calorie ingestion (kcal/ 






day) 


5^6 


26.7 


73,6 


200.0 


86.2 


Slope of growth in hnghi to calorie 




2.9 


«.3 


. 0.1 


1.3 


ingestion (mm/ 10^ kcal)* 




(±2.6) 


(±2.8) 


(±1.7) 


(±6.2) 


Slope of growth in wfighi to calorie 




124 


44 


1 1 


'47 


ingestion (gm/10^ kcal)* 




(±87) , 


(±84) 


(±14) 


(±9) 



* Mean (± iiandard deviation) 



E 19h 

i 18 

i 17 

a 

^ 161 



15l 



± 



50 



100 



200 



Supplemental Calorie Ingestion 
(kcal/day) . 

Figure 2. Growth in hfcight ofchildren ages I to 3 as a function of 
supplemental calorie ingestion 
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C 

m - 
E 

£ ^ 
c ^ 



.0 




200 



Supplemental Calorie Ingestion 
(kckl/day) 

Figure 3. Gromit\ in weight of children ages 1 to 3. as a function of 
supplemental calorie ingestion • 

•calories, controlling for initial size. Cbntrolling for initial size means 
that negligible influences of size) at 1 year of age (correlatioh 
coefficient of wei^t at 1 year of '^ge and subsequent gain in weight is 
r = -0.094; P ^ 0.10; and for height- and subsequent growth in 
height, r = 0.003, P ^ 0.10) which might affect subsequent growth 
have been taken into account and, therefore, size at 1 year of age 
cannot be an explanation for the findings. 

Similarly, one would like to be sure that no factors other than the 
nutrient intake from the supplement could be the cause of the 
relationship between supplement ingestion and growth. Factors which 
influence a person to cooperate with a study could also affect growth 
quite independently of any benefits received from the supplementa- 
tion. To reject this possibility, one must introduce measures of 
cooperation into the energy-to-growth regression and show that they 
do not affect the findings in table 3. 

An unequivocal measure of cooperation with the supplementatipn , 
program is the amount of energy ingested by drinking the suppl'e- 
ment. Infant calorie ingestion from the supplement is associated (r = 
0.72; P ^ 0.01) with calorie ingestion from 1 to 3 years of age and, 
therefore, infant consumption should reflect many of the influences 
which affect later supplement consumption. Including infant calorie 
ingestion from the supplement in the regression relating growth to 
the energy consumption from 1 to 3 years of age liad no influence 
on the energy-to-growth conversion slopes in tabl^^. This was 
reflected by statistically nonsignificant partial correlation coefficients, 
Tp, of infant energy intake from supplementation into the energ^-to- 
growth conversion equation (for growth in weight rp = 0.05, P ^ 
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0,10; for growth in height r -0.(J2. P > 0.10). Orher measures of 
cooperation are the number of days a child attended the supplemen- 
tation-center and the amount of supplement ingested. Introducing 
^ these measures of cooperation separately into the regression analysis^ 
of the jpffect of calorie intake on jjfrowth revealed that neither days 
attended (r ^ = 0,01 for growth in weight, ^ 0,10; rp = 0,06, for 
growth in height, P ^ 0.10) nor amount of supplement ingested (rp» 
= -0,05 for growth io weight. P ^ 0,10; r,, = 0,00 for'^growth in 
height, P S: 0,10) had any significant effect on the slopes in table 3, 
When all these indicators of cooperation wer« entered together into 
the regression, thus. taking them all into account at! once, the rates of 
energy conversion to growth were essentially the same as those shown 
in table 3. Thi^ energy-to-wei^ht gain was 64 gm/10^ kcal (P :S 0,01) , 
and the energy-to-height gain was 1.6 mm/10^ kcal (P :S 0,01) when 
all the above measured indicators of c6operatioh were taken into 
account. Therefore, the significant jeffect of calories on growth could 
not be explained by anything except the nutrients in the supplements 
themselves. >^ 

T«hese analyses assume that there, is no substitution of the supple- 
ments for the home diets. In other words, every calorie ingested fr^m 
tht supplement is a calorie added to the dietary mean at every level 
of supplementary intake. This is in fact almost the case, because the 
dietary intake decreases only 1 calorie for every 10 calories of 
supplerfjent ingested. Such a small dietary replacement effect of the 
supplement does not affect the figures in table 3. v 

Finally, as expected, there tire statistically significant nonlinearities 
in the overall rate (P ^ 0.002), which*is reflected in the differences in 
the slopes with category >)f consumption as shown in figures 2 and 3. 
In other words, the more calories ingested from the supplement, the 
less the efficiency of conversion into growth. 

« 

UTILIZATION OF RESULTS 

The purpose of the analyses was to confirm that the intervention 
used in this study did have an impact on the nutritional status of the 
child, ajT aim which would seem to be achieved.^ We must n<^, 
however that what is being confirmed is that the supplemental food, 
when it is being corisumed, has an impact on the child's nutritional 
status. This is not the same thing as confirming that the intervention 
affected an important number of children. That would require a 
different study (design, one contrasting the villages with other nfonsup- 
plemented villages, which was not done. Even so, the small number 
of children in the highest supplement category makes it clear that-' 
our intervention was not very successful as an effort to t/eat 

V 
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populatiorw, despite its success as a confirmation that an intervention 
had effect?H^deed it gave biologically meaningfal utilization rates 
^which ^were in accofd with our prior notions of diminishing returns, 

DISCUSSION 

Three ^conclu^ns thaf are pertinent to the evaluation of child 
health services may be drawn from our presentatien. *" 

Th« first is already well knQwn: The evaluation of an intervention 
should depend on its purposes, a confirmation of the biological 
effectiveness of the supplementary feeding, the study is a success, 
Wfre this program to be evaluated as a public health intervention xp 
improve the growth rates of the children in the four villages, 
however, it would ^not rate very highly, either absolutely or ^ost- 
effectively. This is not surprising, because the program was fiot 
designjd[ for this purpose. 

The second conclusion is that in the area of nutritional indicators 
the adequacy %vith which the efficaty of an intervention can be 
evahaated is sharply limited by the* state of present knowledge. This 
results il5 uncertainties as to program effectiveness and efficiency. 
Careful design can perhaps in part face this problem, but rpost 
interventions will be practically rather than scientifically designed. 
For. these, the lack of useful information as to what the response 
should be will sever^y limit ^valuation. The lack of.adequate baseline' 
data is of course often a limiting factor as well, but currently the most 
serious limitation to evaluation is the absence of more theory based 
on experimental data from field studies which can be applied to 
make numerical predirtions of response. This study, for* instance, 
revealed a constraint on gjrowth, due to nondietary factors, which 
corresponds in magnitude to the deleterious effect of diarrhea on 
growth (Martorell, Habjcht, Yarbrough, ^t al. 1975a; Martorell, 
Yarbrough, Lechtig, et ah 1975). 

The third^ conclusion is that in many cases there is an inevitable 
tension between- demonstration of an impact of a multifaceted 
j^rvention and the evaluation of its components. The confirmation 
that an agent has had an effect requires either an unambiguous 
P4*ediction. which current theory cannot ^grpvide, or a range of 
variation in the treatment which will lead to less than maximal 
possible impact on the population as a whole. The fact that practical 
interventions tend to introduce simufpineously such variables as 
nutrition, education, medical care^ and economic changes, increases 
the complexity of this problem manyfold. Thus we see the absence of 
^careful theory of nutrition and {physiological response to nutrition 
as a fundamental practical limitation' to the evaluation of programs, 
and not jiist a scientifrr nicety. 
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y , ■ ^ DISCUSSION; • - 

/ ' Maurice Bact^tt and Sarbgrd Starfi^ld • ^ 

' " , . .... - - 

The objective X)f this study was to examine the efft^t bf^a caloric 

' ;^pplement.»on height and" weight of children whose antffra0p^ v 

morphic m^asuremeqts. f;^U belov^ ,staildards . for Nort^ AmpiiCay ^-.^ 
children. As s|ic|;^, it isan effectiveness study because it ekamines lYi 
relationship between, a process of car^ ai|d the-rcsuH of t|iar process^ 
on health status. \/ \ • ' ^ ^ ^ 

THie majop^ limitation of* the' st^dy was its po^ibie iriappropriate- 
n^. The goal of improving height and 'weight f J N^fth American 
standards was not i^ne expressed Ky tjie community, tiut rather one 
d^ived from the researchers. Whether the failure of the rfsearch to- 
"sell" the intervention '(supplementary caloric/fceding) resulted from 
lack of c^mrhu^lwy participation or whetjierlit .^s due l6 other 
factors is Hinclear. Thus the researchers were faced\fith a situation 
wher^the intervention w^s efficacious (it produced the desired result 
under carefully controlled conditions), but not effective (in the 
community setting, it could not be implemented).. i • 

f The jjipplicability of the JntervenfionVas considered inappropriate, 
because the intervention did not appear to be a useful bne in the 
community. There are, howeveu, lessons to be learned gbout the 
method of research. If it is the intention of the researcher that the 
results of a study be appHed, it is necessary tliat the goal be expificitly' 
recognized as legitimate by those for whom it is intended. The goal 
of generalizability in this research was attained at the ej^pense of 
implementabilitV q£ the intervention. The tetter might have been 
realized' by parucipatlWn of the target population in its planning. 
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- / CHILD heal™ and THE COMMV>^: 

THE ROCpESTER CHILD HEAlTH STUDIES, 1967-1977 

: ' -Ik - L B. Pless . . . 

Summary ^ * ^ , ^ 

' ^ , • .. ■ . ■ ' 

The broad goal of these studies is to improve the Jjealth of 511 
children an a defined community; more specific objectives are the 
reduction in mdrtality at all ages and the prevention (both primary 
and secondary) of morbidity from a wide variety of^ common 
childhood disor4ers. To achieve these ambitious kims, new^Klbalth 
services were developed and evaluated and, through health educa- 
tion, attempts were made to (Change behavior in health and illness. 
The principal basis for tliese studies was continuous monitoring of 
the health needs of children and theiV utilizatfen through periodic 
household surveys of a Rppulation sample in a single county in 
upstate New Ybrk. The results of th^se surveys.were utilized by the 
department of pediatrics which* sponsored them, by practitioners in 
the community,* by local health, planners, and by medical educators, 
practitioners, and policy, makers elsewhere. 



Introducti<)!n ^ , . ' ^ " 

The series of stymies conducted in the Rochester region under the 
auspices of the University of Rochester Department of Pe'diatrics 
extended over a period of nearly 10 yeaVs. It is our only exanvple in 
which a university department has conducted 2i progrgpn of research 
as opposed to a specific project or projects. This progr^^m is also special 
because it spanrfed a pcrio^ of many years and was directed toward a 
common jfoal. It thereby provides an exampl.e of some advantages of 
this approach and also illustrates some of its limitations. 

If we agree that the main purpose of research is to bring about 
change, then this and other examples ofVhild health studies must be 
examined ir\ relatipn to the question: T(^hat extent did the effort 
produc#the desired change? A 

209 . • ' 
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in the case of the Rochester studies^ the change in question was th^ 
provision of more.and^)elt^ services. Ultimately they aUo aii^ed at 
the broader, albeit less>c*listic, ^al of impr^ng^the healtlj/of all 
chili^en im ^he commmiity. It is ditRc^lt to explain' how better Ifealth 
outcomes can' be^ related to in)proving th? processes of care, when 
Uiuch current evjclence failstb suRpbrt siichr a relationship^ most 
advanced so$;^ties (Cochrime 1972)'. However, if ohe's definijjpn of 
health is Broad en6ugfcr--a[nd if jbe entire context in ,^hich the 
pi^ef^riam is conceived is also broad enough,; we may. argue- thaty^^ 
community is^;^|althier wVien its health servi<jes are more rati&nal and 
more efficient aJfei deli vet abetter product. This may be tri^^^hether 
or* not tlie health of its citizeiis demonstrates improvement when 
traditional measures of^hy$ical pr ^n^mionatwell-being afe used: ^ 

Purpose of Sttidy * ^ * , - ^ ' 

The impAus for the Cornmunity Child Health Stii(Jies ''Program 
came from Dr. Robert Haggerty when hejrf>ecame chairman Df the 
Department of Pediatrics in 196^^ecause of Dr, Haggerty's commi;- 
m^nt to community pedi2jtTics,Cit wa^^a logical first stejV'foV hrrn to 
initiate studies that would provide bcKh a setvsitive assess^ment of areas 
of unmet needs and a baseline against which future programs could 
be evaluated. He realized that the ^best \3y_to obtain the requy-ed 
information was to find a capable sociologSf with special skills in 
surve^,jW*ethodology. He chose Dr. Klaus Roghmann, who has 
irp-d^btedly been, the methodological backb<jne of the research 
progrim,. - < 

The initia^intention was simply to carry ovit a sample survey of ^11 
households m the county to study the healtfirneeds and utilization of 
services of children in those households. The community, in this and 
most of the; stibsequent studies, was defined as Monroe County — an 
interesting, heterogeneous area surrounding the city of Rochester, in 
upptate New York. In composition, the population — about three 
quarters of a million — is typical bfc^any other communities of similar 
size in^ the United States: However, this community differs from 
others in some aspects of health planning and services. One particular 
feature of services planning that makes it atypical is its long record of 
community participatioft in health planning-, particularly regarding 
hospital beds. Prior to 19^4, a" team of external advisers recom- 
mended that pediatric. hospital beds be redistributed to avoid dupli- 
cation and obtain greater efficiency (Stokes 1965). This recommen- 
dation was implemented when Dr. Haggerty arrived. Several small 
units were closed, and specialized services were distributed among 
the community hospitals and the university hospital. In addition, a 
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close relationship was forged between the practicjng commiinity and 
the department. Clir^ical faculty^ppointments were offered to many 

*of the practjlioners. Subsequently, efforts were m^de^to involve thfem 
in collaborative research. Thf effect oPthis move was to diminish the 
"town/gown" separatio.n that fi«equently a characteristic of 

university communities, thereby increasing the ^likelihood fhat' 
changes Iniyated by tW^ department would be acceptable to those in 

'^clinical pracliA. • - ^ . 

In the remaind^of. this case . pre^ntation I shall outline the 
^metlipds involved in pie surveys, describe in general terms^wha^ the 
orogram oP research enc<)mpassed, illustrate sorrte of /the major 
iihdings, and indicate the extent to which they appeaf*to"have ^en^ 
iihplemeqted. / " ' ^ 

Areas o£ Child Health Care \ 

/ . x ■ 

Tms series of studies has dealt'broadly with the entire spectrum of 
childhood morbidity and health -services proyidecH<n^ children in one 
geographfc^ly defined area. Some aspects have recefved more atten- 
tion than others: organizatio'm and utilization of services, school 
problems, behavioral^rfiisorders, chronic illness, and^ealth education. 
The focus emphatically has not been on a single problem, but on the 
relauonship "between several common problems in child health and 
the care provided. > ^ 

Target Population * ^ 

Over the first 7 years of the study, the child' populatioj^j^of the 
country rose from 220,000. to 250,000, with the num'ber of black, 
predominantly poor, children increasing from 37,000 to 65,000. As 
with many similar American communities there was a continuous 
migration of whites toward the suburbs, with blacks remaining in the 
city. Within the city, two poverty areas, definecj both by census tracts 
and by' sociopolitical subdivisions (or wards), were of par^cular 
interest. Here the black pppylation rose from 61 percent to 77 
percent in the 5-year period ^ginning in 1966. Also of interest was 
the western section of the county, to which about 1,000 migrant farm 
workers came annually, often with their children. 

Study Design 

The most coflnmofT methodology utilized throughout these studies 
is the sample survey iWirwick and Lininger 1975, pp. 46-68). In 
•1967, 1969, 1971, and again in 1975, a sample varying between 1 
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percent and 0.5 percent of all hpuseholds in the county was studied- 
systematically. These household surveys focused primarily on chil- 
xh^n but varied in their enffthasis.* Generally, they dealt wrth^/he 
ffW^uency of symptoms as an expression of health needs and then 
examined patterns qf utilization. The variations' /included specific 
sections dealing wrth attitudes toward health care;( the use, in one 
instance, of a 28-day health cakndar or diary for recording stressful 
•events and medical care utilizatipn; school health; behavioral prob- 
lems; and a series oF studies on the care of children with chronic 
illnesses. V 

The core questions inSeach of the surveys were sufficiently. similar 
to allow comparisons to be made over time. Accordingly, conclusions 
could be drawn about the effect of changes that hacf been introduced, 
.either by the program itself, or by local or national events. Further- 
more, to evaluate specific programs such as the neighborhood healdi 
center <NHC), various sections of the community were oversampled 
and modifications in the design were made or experiments conducted 
as the need arose. 

-The random sample and the household interview were used 
throughout. The interview contained questions on illness or ii^ry 
sustained in .the preceding 2 weeks, the use of prescribed" and 
ncM^rescribed medications, the use of medical care servic^ (during a - 
2-w^ and a 12-month .interval), hospitalization, the presence of 
long-t6rn1 or major illness and, in the most recent^^rsion, questions 
about ^^roductive behavior and attitudes. (Earler versions also 
contained a series of questions on attitudes towards illness and 
doctors.) \ 

A num^r of related studies on specific topics were built around 
■ )^on core. Figure 1 presents a grossly simplified picture of • 
^ or first generation, studies. The lower part shows the 
irrent or subsequent substudies during the first 5-year period. 
Figure 2 illustrates the program of research during the second 5-year 
span. Initially this work was supported by the U.S. Children's Bureau 
(Grants H-104and 148) and subsequently by the National Center for 
Health Services and Research Development (USPHS Grants HS-467 
hnd HS-294). 

The bulk of the work was carried out from the Rochester Child 
Health Studies (RCHS) offices— a division which operated as a major 
component of the Department of Pediatrics research program. Dr. 
Haggerty was the principal investigator; Dr. R^oghmann and I and 
three other senior faculty served as project directors. The team also 
included full-time research assistants, two field supervisors, many " 
part-time interviewers and coders, and two full-time programtrs. 

The annual budget for the studies during their peak years averaged 
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Figure 2. Rochester child healUytudies. Phase II: 1972-77 ^ 

at least 1^00,000. An additional $200,000 or more was spent in 
support ${ the clinical training program aimed at residents and 
fellows in ambulatory pediatrics. The average direct cost per com- 
pleted household interview ranged from about $10 to approximately 
$40. Five or six projects were concurrendy under way at any given 
time during a typical year. This was only possible because the senior 
investigators were full-time workers and had a staff devjited exclu- 
sively to these projects. Under other circumstances such as heavier 
clinical conr^mitmenis or additional administrative duties, the output 
would have been much smaller. 

The program relied heavily on computer technology. During the 
latter years four programers were needed to carry the load. The 
major subsets of the studies, directly related to the principal theme, 
fell into several distinct areas: new manpower studies, particularly 
concerning nurse practitioners; a variety of other collaborative re- 
search efforts involving physicians in practice; studies of organiza- 
tional change, such as the NHC; and finally, studies of the care of 
children with chronic illnesses, in addition, attempts were made to 
develop a model fpr evaluating health education programs presented 
in a pediatric setting. 

Many of the studies were done in coUaborafion with nurses, other 
health practitioners, sociologists, psychologists, administrators, and 
health planners. They frequently provided an essential resource for 
colleagues with purely biological interests, such as the study of alpha- 



Rochester Child Health Studies 215 

1 antitrypsin deficiency, genetic disorders amenable to amnioceritesis, 
hemophilus influenza vaccine, or the etiology of allergic disorders, 

Methodology * 

The surveys, by means of standardized interviews and random 
samples of families with children, provided the bulk of the informa- 
tion obtained. TheyVere conceived as a series of linked interrelated 
studies based on the strategy of developing a broad system of data 
collection; the results ivere regro'bped in order to answer a wide 
range of questions. Using the intermittent sequence of cross-sectional 
studies, we could measure the effects of both planned and unplanned 
changes in health care. 

A major design problem jn this type of research may arise when 
one wishes to gefieralize from the findings. Information obtained in 
the interviews with , the 3,000 families was. supplemented by data 
available in Medicaid and Blue Cross insurance payment (or claims) 
files, birth and death statistics, manpower inventories, and census 
summary tapes. In the latter years, we also relied increasingly on 
interviews or questionnaires from physicians in practice. Together 
the data provide a picture of the total system. The community itself, 
however, is both typical and atypical in many respects, and^ this must 
be taken into account when Applying the results elsewhere. 

Although the major surveys do not tell the complete story of this 
program, they merit additional details^In the 1967 survey theVe were 
no available •sampling frames of chHdren apart from the annual 
school census conducted exclusively in the suburban districts. There- 
fore, in the city, a list of 100,000 households was made and four ^ 
sequential samples were drawn, using||^very 3,000th address as a,^ 
starting point of a 10-residence segment* ^pfcluster. Each home was 
visited to see if any children resided there. A maximum^ of two 
children were selected randomly from among those available in the 
home. The^ suburban list permitted a simpler approach, using every 
200th child. Thus the city procedure yielded an unbiased, family 
sample, whereas the suburban approach gav^ an unbiased child ' 
sample; hence correction procedures were needed in the analysis 
(Roghmann and Haggerty 1970). Because only two children were 
selected, a further sampling variation was introduced, since the 
fraction necessarily varied' from family to family. The 1967 survey 
resulted in about 94 percent completed interviews at an^average cost 
of $10 per interview and covered a total of 1.402 children. 

The 1969 survey was a smaller midpoint study (0.5 percent of all 
families with children under age 18). Some groups were oversampled 
(1 percent) to allow for special studies of particular interest. In this 
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study, Blue Cross and Medicaid lists were used as a sampling frame, 
In addition, we included all pediatric admissions to the largest area 
hospital who did not appear on either list. Systematic random 
sampling proceeded until 512 families were selected. By usfrtg an 
abbreviated child questionnaire, we could obtain information about 
all children ioi^ach f^^mily, although most analyses were Confined to 
the mother and hfcr youngest child, "Each mother kept a 28-day 
health calencfer (or diary) to record .stress, medications, and symptoms 
(Roghmann and Haggerty 1972). Eighty-two percent of the interviews 
were completed in spite of the increased demands made on respond-, 
ents. The average cost per interview was $40 because of extra travel, 
coding, and the payment of a nominal sum to each respondent for 
completing the healtirtliary. 

The 1971 survey was similar to that in 1967, but the sampling 
made use of the housing unit li^ frorp the 1970 census. Five 1-in- 
1,000 systematic random sample clusters of addresses were selected 
as starting points, actual names and addresses being obtained from 
the computerized file of a local utility company, A total of 1,216 
families with 2,95^ children were interviewed; an 82 percent coniple- 
*tion rate was recorded at ah avei^e co,st of $13 per family interview. 

The 1975 survey was again similar but was aimed at only 750 
families with approximately 1,800 children. 

Data and Information Systems , 

The results obtained were of many kinds and were interpreted in 
several different ways. The model of the community health system 
shown in figure 3 may help categorize them in a conceptually 
meaningful fashion (Haggerty, Roghmann. and Pless 1975, p. 8).*The' 
central issue is the ability of the care system to interact with health 
needs of the population so as to result in appropriate utilization, 
leading to compliance and hence to improved health. Accordingly, 
data were obtained about some aspects of/ social background arid risk 
factors, such as stress and genetic abnormalities, health status (as 
assessed by disease,- disability, and discomfort), perceived needs, and 
actual demands for care. We also assembled data about various health 
mghpowe?, resources, health agencies, their organizatipnal 'forms, aind 
the manner in which services were made available to those who 
required them. 

Indicators and Measurement Criteria 

Both simple and comple^x criteria were used. Sinvple criteria 
included birth and death rates and the prevalence of varrous 
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disorders or conditions. For the mosi part/parents' reports that a 
- disease or symptom was present were accepted at face value without 
confirmation from a physician. The use of complex indicators was 
usually a result of being faced with the need to operationalize some 
abstract or imprecise concept. Examples oT the latter were health 
supervision (Hockelman and Peters 1972). maladjustment (Pless, 
koghmann. and Haggerty 1972). family functioning? (Pless and 
Satterwhite 1973), total family impact, stress, and crisis (Roghmann 
and Haggerty 1973). In each case, various approaches to index 
information were employed, such as aggregating several indicators 
with or without simple weighting, and only occasiOrlally using factor 
analysis or other statistical techniques to generate weights. At no 
point did we attempt to deal with the formidable challenge of 
attempting to. develop a child health status index for the whole 
population, althoi^h such a measure would clearly have been of 
immense value in assessing the total impact of the program. 

Type of Evaluation ^ 

Virtually the entire repertoir^ of evaluation strategies was used. At 
one stage in studying the utilization of services, both pj-ovider avid 
survey data were linked to inputs and health manpower resources. 
Efficacy, effectiveness, and to a l|sser extent, efficiency were consid- 
ered in the studies of the health center, the nurse practitioner, the 
family counselor, and several other programs of care. Examples of 
cost-benefit analyses, in the traditional sense, were isolated. The few 
direct attempts at measuring^'*quality*.' Were bas^d on assessment* of 
short-term outcomes. However, for the most part, the primary 
objectives were directed more toward assessing quantitative aspects of 
the delivery of health services. Of interest in this regard are t^e 
findings that, based on the number of medical care contacts, utiliza- 
tion of ambulatory care service* declined over the period from 1967 
to 1971 and similarly, hospital inpatient occupancy rates fell steadily. 

Results J ^ 

The main results of these s<udies are described in a book. Child 
Health and the Community (Haggerty. Roghmann, and Pless 1975). as 
well as in approximately 50 scientific publications. Since it is impossi- 
ble to do justice to the wide range and depth of findings obtained, 
only a few of the more important have been selected for presentation 
^ in this section. 

In the fir^t 5-year period the following results wefe achieved: 
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—The survcyr'cstimated the prevalence of unmet medical n^eds of 
children, and the uneven distribution of services inlLhe community 
were documented with reasonable precision (Haggerty 1970). 

— 'the : household "Surveys revealed the dimension^ of the "new 
childhood morbidity" tHaggerty, Roghmahn, and Pless 1975, pp. 

94-113). ; 

—Health and illness behavior were related to family crises, with stress 
of various kinds increasing utilization of health services, provided 
that the site-of care was easily accessible. If barriers such as 

f appointment systems vyere imposed, stress appeared actually to 
decrease utilization; in both situations the effects differed in 

•/ mothers and children (Haggerty, Roghtnann, and Pless 1975, pp. 
142-154). 

—An experimental study demonstrated the effectiveness of the 
pediatric, nurse practitioner in the office setting, along with the 
acceptability, financial feasibility, and impact on quality of care of 
this new form of health manpower (Charney and Kiuman 1971). 
— The siirvey documented the 'distribution of chronic illnes^s among 
children and the fr$?queney of .setondary psycho.social dysfunc- 
tion, particularly as a reflection of famfly functioning (Pless and 
Roghniann 1971). A regional survey of physicians in. primary care 
who must deal with these children revealed several critical problem 
areas (Pless, S'atterwhite, and Van Vechten 1976). 

— Methods were develrtoed for identifying behavioral problems in 
^ children age 2, and fhe problems were correlated with parenting 
techniques (Chamberlin Itty^; A controlled trial to test hypotheses 
about the reinforcement pWuch problems by inappropriate paren- 
tal responses was initiated. 

— Studies of suburban adolescents showed the need for speciali^^eci 
counseling programs. ^ ^ -''^^ 

—Controlled studies of the NHC showed a decrdSse in hospitalization 
(Klein et al. 1973) and emergency room use for those registered 
(Hochheiser, Woodward, and Charney 1971). 

— Studies following the introduction of Medicaid in New York State 
showed few changes in the place, frequency, or patterns of care for 
the poor and little basic alteration in thje pre-existip^ "two-class" 
system of care (Roghmann, Haggerty, and Lorenz/1^71). 

— The continuous data \tdse permitted the eff^fof change in the, 
New York State abortion law to be studied^Rowing a decline in 
total births andsin births of high-risk babies, thus reducing the 
incidence of prematurity and infant mortality (Roghmann 1975). 

— The collaborative research program assessed the value of white 
blood cell counts in predicting the results of treatment of children 

. with pneumonia (Shuttleworth and, Charney 1971); it examined^the 
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correlation of clinical and other fjictors related to compliance with 
oral pemcillin prescriptions (Chafney et al. 1967); later it showed. 
^, the effectiveness of using long-term sulfa in the treatment of 
recurrent otitis media, again through a controlled; double-blind 
experimental trial (Perrin etal. 1974). 

Utilizadbn of Results 

In an attempt to evaluate our progress, we decide^! to determine 
the extent to which two of the program's publications (Perrin et al 
1974. Kanthor et al. 1974) influenced the. thinking and behavior of 
^ pracutioners m the community. All practitioners received a question- 
.nairp about their awareness of the results of these studies (both had 
^*een tJubluhed in reputable journals) and the extent to which those 
wh6 had read them were prepared to implement the major recom- 
mendations. Fewer- than half of the morp than 70 practitioners 
surveyed were both aware of and implementing the results of either 
or both studies. Thus the second stage involved either a letter of 
encouragement from the department chairman, along with a copy of 
the reprint in question, or personal visits by our directol- of commu, 
mcauons to a second, randomly selected portion of the group. A 
follow-up questionnaire showed only a small increase ia reported 
Jntentions to adopt the recommendations, with more of those who 
had been visited willing to do so than those whoMd not. 

Thesew conclusions, which suggest that more, than publication in 
widely read journals is needed to persuade physicians to utilize the 
findings of medical research, must also be interpreted as evidence of 
how little we know about other, more successful, methods of persua- 
-sion. They indicate the problem at the "micro" level, that is within 
the immediate target community. It is conceivable that greater impact 
may have occurred elsewhere within the scientific community or 
amdng practiUoners in other parts of the country. Clearly, however 
we kno^ little about jhe factors that ^fluence the diffusion of new 
Ideas, whether or noAey stem from research findings. As investiga^ 
tors, we appear to caWlittle about this problem. Perhaps there is no 
way of knowmg what influence any of the work in Rochester has had 
on the thinkmg of others. Counting reprint requests is a crude and 
unreliable indicator of anything other than the popularity of a title or 
a particular subject at a given time. 

Upon reflection, we suspect that there are a number of .areas in 
which the impact of this program was probably significant in spite of 
the pessimism r<;flected in the preceding section. One such area is the 
introduction of the idea that there is a new morbidity in North 
American pediatrics (Haggerty 1968). We were not the first to call 
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attention to the prominence of such common pr6blems'i|9s school or 
behavioral disorders, or tlie difficulties related to the management of» 
the chronically ill and handicapped; however, our having coined and 
popularized the term "the new moVbidity*' appears to have persuaded 
others to address these issues more in^nsively. 

Secondly, the work in Rochflkr, combined with that of others 
elsewhere in the area of health manpower (particularly the^work of 
Dr. Charney in the nurse practitioner. studies (Charney and Kitzman 
1971) and the use of nonprofessional family counselors to assist in 
the care of the chronically ill (Pless and Satterwhite 1972), appears to 
have also aroused considerable interest. 

Tfiird. at a time wl\en the valtie of these centers was still debatable;, 
the work demonstrating the worth of the NHC has undojubtedly 
influenced the thinking of others. 

Fourth, drawing attention to the special needs'of the chronically ill, 
with respect to possibilities for prevention of secondary psychological 
and psychosocial problems, appears to have focused increasing 
attention on this previously neglected area. 

finally, there was the achievement of the most important, but least 
explicit, objective: to provide an example to other pediatric depart- 
ments of how an academic department could develop a scientific 
program in community pediatrics. This, result is the most perplexing 
and least cleai;-cut. On the one hand, it appears from an analysis of 
abstracts submitted to various public health or pediatric societies that 
this department has incieed played a leading role in the evolution of 
cpmmunity pediatrics. More aitdWlore graduates are -seeking fellow- 
ship level training in this specialty and it is reasonable to assume that 
this growing interest was stimulated in part by the achievements^ of 
the Rochester program. On the other hand, no other department has 
tried to emulate this exAiple with respect to its research commitment. 
Whether this should be interpreted as an indication that, in their 
judgment, the effort is not worthwhile, or whether they lack the 
resources, Hnancial or personal, we cannot say. 

At the clinical l^vel, it seems unlikely that these efforts have 
substantially improved the health of children in the county itself, let 
alone that of children elsewhere. It has, however, provided some 
important tools for doing so and others for assessing changes in 
health and health care research methodology. It demonstrated that 
with ingenuity, effort, aind some skill, it is possible to apply the 
randomized clinical trial to many unconventional problems. It dem- 
onstrated the usefulness of the 28-day calendar in studying relation- 
ships between stress and utilization of health services. This approach 
helped clarify spme of the basic issues in utilization, models such as 
those proposed by other investigators: Suchman (1967), Andersen 
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(1968), and Rosenstock (1966)/As a, research tool for assessing the 
quality oj:>f'amily life, the FainlTy Functioning Index (FFI) may have 
clinical applications as well (Pless and Saiterwhite 1973), The feasibil- 
ity of using Medicaid and Blue CrosI data tapes for the analysis of 
utilization patterns over time and between groups has been demon- 
strated. Several basic problems in methodology involved in sampling 
children in households have also been clarified. 

Toward the end of the 10-year period the program has increasingly 
influenced local health planners. At least some of the ideas in the 
region are indirectly a product of data -produced by the RCHS, The 
collaborative research program continues to fldirish. and increasing 
numbers of praoiumiei-s are becoming involved in clinical projects 
that draw upon the expertise of the investigators in the RCHS ^oup. 

At the nationaF leveljt seems quite likely the, papers dealing with 
the supposed "medical care crisis" (based-tjn an analysis of utilization 
data), the effects of the Medicaid program on care patterns for 
children, and the effects of the health center have ^11 had an impact 
on the thinking of those responsible for policy formation, 

■' ' . ' I . ■ 

Discussion 

This program of evaluation of child health care differs from other 
case studies presented at this conference in several important re- 
spects: In the first place, it is based exclusively on a defined 
community; second, it has originated frpm an academic department; 
and third, it has attempted to answer simultaneously both theoretical 
and practical questions (table 1). How well it has succeeded in 
meeting these potentially conflicting goals is a Question of some 
importance for a conference dedicated to examining the interface 
between research and medical practice. 

This work is based on a belief that academic departments have an . 
obligation to deal with issues of concern^ to the community which the 
university has been created to serve. It may be argued that in such a 
setting the sole commitment is to the "community of scholars." In>this 
case, answering basic, fundamental (as opposed to "applied") ques- 
tions is all that matters, and this orientation prevails in most areas of 
academic endeavqr. However, in the field of health care research, 
specifically community pediatrics— a vital, rapidly developing disci- 
pline — this ivory tower approach cannot succeed. 

Most, questions relevance. to health care investigators must be 
posed in real-life terms. They cannot be divorced from the ultrmate 
test of the likelihood that the solutions they yield .will be applied. 
Recognizing this issiip. however, does not make the task any easier. If 
anything, it heightens the dilemma that confronts those who. from an 

. ■ ■ ■ I- 
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Table 1. Guiding Principles of the Rochester Program of Child Health 
Research 

1 . Focusing of studies on defined populations (sampling permits generalizaubns) 

2. Us^^f experfft(iental and quasi-experimental designs whenever possible to' permit 



"cau^**Jnferekices ' 7 

3. Linking of theoretical and. Hi/^thodologic studies to practical i^oblems to permit 
"relevance" and "science" to co-exist ^* ^ r 

4. Linking of research to teaching and service to enhance accemability and long-term 
benefits y 

5. Combination of sociomedical and biomedical aspects to.enhimce value of each 

6. 'Application of results to solution of f>roblem$ at many levels. 



academic perspective, try to address problems whose ultimate solution 
depends only in part (and perhaps in. very small part) on the 
acquisition of knowledge alone. Clearly social, economic, political, 
and other forces are at work. What is learned from research must 
interact .with these Torces before any signififcant. changes in policies 
can be expected. While many of us who engage in this type of 
research have expertise in the techniques Required to achieve scientifo 
ically respectable results, few have the skills, knowledge, ox..jevcn 
understanding of the complicated processes whereby social change is 
brqught about. We are therefore operating in' a peculiar vacuum — 
one that may or may hot be unique to this •field of investigajtion. 
" Even at a more basic level of simply diffusing and implementing 
new ideas which niay be df value to those in clinical practice, our 
knowledge and sl^s are woefully deficient. It has been estimated 
that between 50 percent and «75 percent of clinical advances are never 
implemented.. The exact figure is unimportant; what. is significant-is 
the increasing. realization by funding agencies, as well as the public, 
that we must becpmre more accountable for research expenditures. 
Investigators need to recognize the limitations in their ability to 
"market** research results and begin to try to rectify these shortcom- 
ings. More energy must be devoted to examining this problem. Part 
of the explanation may lie in the fact that most studies originate from 
social scientists and tend to examine the diffusion and implementa- 
tion of research findings in a theoretical rather than a practical vein.. 

It is difffclilt to con-vey fully the degree of interdigitation that has 
existed between this research program, the department as a whole, 
and the community. These studies have played a central role in the 
depiartment, particularly with respect to its research and ambulatory 
care teaching programs. This aspect makes the RCHS so unusual and 
hence so important as a model for other clinical departments. 

In doing so,>we must be candid about some of the problems 
associated with operating a program of this kind in a university 
setting. It is probably fair to say that many of our colleagues in other 
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clinical disciplined either could not understand, or were skeptical 
about, the emphasis placed on the RCHS, Many may even have 
resented the support it received, both internally and externally, and 
the opportunity this support gave some of us to engage in research- 
on a full-time basis. Coupled with these internal problems were other, 
more basic, questions about the place of such a community-oriented 
applied research program in a clinical department. The university 
itself questioned the involvement of any part of the medical school in 
developing programs of care beyond those essentia to the fundan^en- 
tal purpose of training physicians or nurses. We counterargued that 
it is only by developing programs in the community that we provide 
realistic models for future students. And only through research which 
examines those models scientifically will disciplines such as commu- 
nity medicine and epidemiology gain useful knowledge and wide- 
spread relevance. 

The fundamental belief that a department of pediatrics must be 
actively engaged in the develppment and evaluation of community 
programs remains unchanged. We hope that many colleagues now 
agree. Unfortunately, however, only a few other departments of 
pediatrics in North America have followed the Rochester example. 

Conclusion 

In summary, it should be clear that this pr/)gram has dealt not with 
one study but with many, all of which relate to a central theme. One 
overriding purpose of this conference is to examine child health 
evaluation methods that are applicable in different settings. Based on 
our experience, there seems no doubt that heavy reliance on the 
survey approach provides at least part of the answer. Undoubtedly, 
thoughtful design of surveys allowing an examination of effects over 
time or the establishment of appropriate corrtrol groups, permits 
conclusions to be drawn about how health care changes or how the 
health of children is affected by that care! It is equally apparent, 
however^ that no single approach can deal adequately with the many . 
diverse issues encompassed by the term '*child health." A universal 
health status index for children may ultimately be of value in work of 
this kind, but efforts thus far have shown this to be extraordinarily 
difficult to conceptualize, let alone put into operation. For the time 
being, therefore, we conclude that to evaluate child health adequately 
we must employ many different philosophies and approaches, con- 
stantly attempting to unify them under a single Conceptual frame- 
wwk. 
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DISCUSSION 
David Bersh and Carlos V, Serrano 

The Rochester evaluation program consisted of a series of studies 
conducted between 1967 and 1977. Evaluation, however, was merely 
a single component of a broader demonstration program whose 
overall objectives were to determine hpw community pediatrics-\^ 
including cpmmunity-based research and health planning— could be 
introduced into a clinical department of pediatrtcs, and to evaluate 
the impact of such a program on the community and, in particular, 
on the health pf children. ^ 

The evaluation component depended primarily on data collected 
through a series of home interviews supplemented with data from 
other sources. The program itself did not develop original evaluation 
tools, but rather applied and integrated the data resulting from a 
wide variety of previously existing, validated procedures and metho- 
dologies. There was an attempt to make appropriate use of the 
collected data to improve health status, to asses3 the. internal success 
of the program's components, and to determine the ability of the 
health sector to comply with health needs. \ 

A number of valuable studies were carried out on various subjects, 
such as unmet health needs, effectiveness and acceptability of the 
pediatric health practitioner, and chronic illness artiong children. The 
investigators were satisfied with the outcome of the internal program 
evaluation; however, questions were raised regarding the long-term 
impact of the endeavor. Although medjcal students were particularly 
attracted to the^rogram, members of other clinical departments did 
^not understand it, or vyere skeptical and resistant to the program and 
the importance it attached to community activities and community/ 
consumer perceptions. No other pediatrics department in the United 
States has committed itself to community involvement in such a 
systematic way, and even the Rochester program has not been 
continued beyond the demonstration phase. 

Additionally, the Rochester program faced tertain problems which, 
would make replication difficult. The cost (approximately $.10 to $40 
per interview) was obviously quite high, although it was noted that 
cost would present less of a problem in Latin America, where suryeys 
can be conducted at less expens^e through the participation of 
students. It was also difficult for the program to maintain ideological 
and methodological continuity over a 10-year period, because of 
normal turnover in professional staff. Finally, the community did not 
participate in the program in any significant way, inasmuch as funds 
were provided through the medical schgol rather than through 
community organizations. . ^ 
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Despite these caveats, the meeting consensus was that the final 
iAipact of the program in terms of education and service will not 
become apparent until after those students who were exposed to it 
are in a position to affect the delivery of pediatric care. The 
workshops considered the Rochester program a classic example of 
applied research, providing a realistic and invaluable learning expe- 
rience for future doctors, nurses, and other members of the health 
team/ 
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HEALTH ACCOUNTING: AN OUTCOME^BASED 
S'TRATEGY OF HEALTH CARE QUALITY ASSURANCE 

]ohn W, Wlliamson 

^ X_§unuxiary 

This paper describes an outcome-based strategy for integrating the 
functions of health care quality assessment and improvement called 
"health accounting." This approach depends upon skillful use of 
small group judgment methods in a five-stage procedure that begins 
with a structured method for identifying cost-effective topics for 
study (stage I), and concludes by reassessing outcomes to document 
impact in terms of improvement (stage 5). ^ - ■ " 

This approach is illustrated here by I of the 74\health care outcome 
studies that have been implemented in the pAst 14 years by the 
author in multispeciaUy group clinics and/or their .associated hospitals 
throughout the United States. , f 

Finally, the results of extensive utilization by other investigators 
£ and intensive reliability and validity evaluation tests of health account- 
ing by the author and his colleagues have establishtd its feasibility 
and provided substantial supportive evidence for recommending its 
more widespread applicatiori. v 



Purpose of Snidy 




TKe foUotving demonstrates an 'outcome-based strategy for assess- 
ing aocMm proving the provision of health care. The concepts 
encompassed were developed as part of the Health Accounting 
Project, which has evolved'Over the past 14 years in 23 multispecialty 
group clinics and/or their associated hospitals throughout ^he United 
States. During this time, over 277 study topics have been formally 
proposed, and 74 outcome-ba«ed health care studies have been 
completed. 

This work represents the author's personal initiative. It has been 
financially supported by grants and contracts from the National 
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Center for Health Services Research in the U.S. Department of 
Health, Education, and Welfare: Private ^foundations, such as the 
Commonwealth Fund and the Milbank Fund, have also made major 
contributions. The results of this research and development have had 
direct influence on the policy development of qu4lity assurance 
activity in the United Statesr. namely the major strategies of the 
Professional Standards Review Organization (PSRO) and the quality 
assurance procedures of the Joint Commission for Accreditation of 
Hospitals (JCAH), 

This paper outlines the project, as illustrated by data ^rom the 
hypertension studies in two health maintenance organizations 
(HMOs) in Los Angeles. California. Next, the in/luence of this 
approach and its application by other investigators are summarized. 
Finally, the advantages and disadvantages (6f health accounting 
strategy are briefly enumerated. ^ 

Area of Health Care 

The topic of hypertension was selected because it was one of th^ 
earliest studied and. provides the most dramatic illustration of the 
total health accounting strategy. Hypertension is a common hea|th 
problem with disastrous results in inadequately managed patients. 
Abundant evidence has indicated that nearly half the patients with 
tliis problem have not been diagnosed, and about half the ones who 
have been detected are not buing controlled. Consequently, approxi- 
mately three out of four hypertensive patients are presently at high 
risk for serious complications including stroke, heart failure, myocar- 
dial infarction, or kidney disease. Finally, recent evidence in^dicates 
that hypertension may also be a seriously neglected health problem in 
children (Heyden et al. 1969)» 

The seridv>sness of the hypertensive condition has been previously 
illustrated by the author in a study of an emergency room walk-in 
population in a city hospital affilbted wirti the Johns Hopkins School 
of Medicine (Williamson 1,971J/A 1-year follow-up of a sample of 
hypertensive working class patients whose average age was 49 
revealed that 1 in 5 (20 percent of the study sample) had died (figure 
^^1). compared to the maximum acceptable outcome assessment stand- 
ard of 10 percent set by the medjcal staff in this institution. 
Subsequent study revealed that three out of four of these deaths Vere 
probably preventable, since they were directly related to inadequate 
patient compliance with required medical management of hyperten- 
sion. ._. > 

A subsequent partial replication of this study by Brook (1975^) 
confirmed these findings (figure 2). The blood pressure control of 
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Figure 1. Case-fatality rate for hypertension found'in a city hQ>pi(al affiliated 
with Johns Hopkins University in 1966. jDne-year followup-by Williamson 
of 87 patients [From Williamson ei al. 1975. Reproduced with ihe^ 
permission of the Bulletin of the New Y'ork Academy of Medicine, Further 
reproduction prohibited without permission of copyright holder.] 



treated and untreated patients was not significantly different. Again,* 
patient compliance seemed to be the major factor involved in lack of 
health improvement. <' • # 

In asubs<^uent study, Inui (1976) demonstrated that il is possible 
to improve patient health outcomes by instructing physipans ti> 
provide more effective health education to patients: FigiiVe 3 indicates, 
the results of a controlled trial in which one set of physicians was 
given focused tutorial instruction on health education of their own 
patient's. A matched set of pHysicians was given placebo educatior/in 
didactic hyperteiision facts. Because of such findings, hypertension 
was selected as one of the best topics to demonstrate the health 
accounting strategy in the following section of this report.- * 

Stud^rpesign ' 

Health accounting is a five-stage cyclic strategy emphasizing health 
outcomes, as is illustrated in -figure 4. The first stage consists of the. 
fojynulation of priorities for (quality assurance topic selection by a 
tesup appointed from those "most familiar with the clinic or hospital. 
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Figure 2. Uncontrolled blood pressure among an independent cohort of 105 
• hypertensive patients in 1971.' Five-month followup by Brook. There was 
Jio statistiiially significant difference in blood pressure comrol between the 
groups. (City hospital affiliated with Johns Hopkins University) [From 
Williamson et a|. 1975. Reproduced with the permission of the Bulletin of 
the New York Academy of Medicine. J'urther reproduction prohibited without 
permission of copyright holder ] 
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Figure 3. Uncontrolled blood pressure among 102 hypertensive patients in 
medical clinics of Johns Hopkins University in 1972 with and without the 
education of physicians. Two month followup by Inui [From Williamson et 
al. 197,5. Reproduced with the permission of the Bulletin of the New York 
Academy of Medicine^. Further reproduction prohibited without permission 
of copyright holder.] 
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Figure 4. Stages of the Health Accounting Project in its strategy to assure an 
approved Outcome [From Williamson et al. 1975. Reproduced with the 

. permission of the Bulletin of the New' York Academy of Medicine, Further 
reprQ(iuc0on prohibited without permission of copyright holder.] 

Using formal^i^focedures for small group estimates, a priority-setting 
team selects those topics that have the most potential for improving 
the health of patients or reducing expenditures for medical care. 

The second stage initially assesses the outcome of medical care for 
the chosen topic. This assessment is designed by a second team of 
qualified clinic staff members with the ai<J of -an outside specialist in 
a field relevant to the chosen topic. The assessment design that is 
developed incorporates explicit estimates of the total benefit to health 
which can be causally related to tht diagnostic and therapeutic 
medical intervention used at that facility. These estimates provide a* 
basis for predicting the potential health impact of the study; they also 
provide standards for evaluating the outcome. Measures of the health 
results in patients ai^e obtained in follow-up studies by an evaluation . 
assistant called a *^ealth accountant.'^^The health accountant can be a 
health profcssfonal, such as a nurse, or merely a high scho6l graduate 
with perhaps 2 years of college, who has demonstrated the problem- * 
solving skills and personality characteristics required to gather data 
and interview patients. The results measured by the health accountant 
are compared with the standards previously established by the team. 
If serious discrepancies are found, further .action in the subsequent 
stages of the strategy is usually recommended. 

The third stage consists of tiiore definitive evaluation Studies by a 
variety of relevant methods to identify correctable determinant^ of 
the unsatisfactory outcomes. Again, the design is provided by the 
study team staff meipbers, and the measures are made by the health 
accountant. 

The fourth sta^^^cdnsists of a replication of the original assessment 
of results in patients to determine whether health standards have 'now 
been metv If not, stages 3, 4, and 5 are repeated one or more times 
until. Acceptable improvement has, been achieved or until it is clear 
that any further gain will not be worth the effort required. 
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Study Results 

In 1972, an independent health accounting system of quality 
assurance wds organized by the California Medical Group, a subsidi- 
ary of HMO International in Los Angeles. The following report 
describes part of their 'experience with this approach in 2 of their 21 
clinics; the results of studies in these clinics have been combined in 
this presentation. 

^ During stage 1, a team of physicians and administrative personnel 
was organized to identify those assessment topics in which quality 
assurance activity might produce the mo.st improvement In the health 
,of chnic patients. Hypertension was one of the five topics identified. 

In stage 2, initial assessments^ of the results, of hypertension 
treatment were made. The clinical study team decided that it was 
unacceptable for more than 5 percent of their hypertensive patients 
to have uncontrolled high blood pressure. Figure 5 indicates that the 
finding of 36 percent of patients with uncontrolled higfh blood 
pressure was seriously deficient for the 248 consecutive walk-in 
patients found to have hypertension in a 1-month sample. 

During stage 3 the determinants of the deficient outcomes were 
analyzed more diefinitively by means of two questionnaires adminis- 
tered by the health accountants: One questionnaire was directed 
toward the physician, and the other focused on the patient. The 
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Figure 5. Uncontrolled blood pressure found^tti the initial assessment of 248 
patients participating in the study by HMO Intet^iiktional fFrom Williamson 
et al. 197.5. Reproduced with the permission <^if^tjm^BulleTm of the New York 
Academy of Afedicine. Further reproduction g^i^mbiied without permission 
of copyright holdec]. 
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Table 1. Hypertension Questionnaire of Physicians on Knowledge Items 



Inadequate drug information ' ywaans 
Unaware danger ofliypertension not related to symptoms 4 
Overestimated naUonal control of hypertension and compliance 8 
Overestimated control of hypertension or compliance by his ownc 
patient* 



10 



Total with inad equate information^ 



11 



* Counting each ixhysidan only once. 

questions were few and were concentrated oh those factors for which 
the efficacy of medical care interventions for essential hypertension 
was documented and established. Tabic I illustrates the results of 
some of the knowledge items on the physicians' questionnaire. All 14 
physician^ queried had adequate textbook knowledge of jiypertension 
pharmaceutics; all were well informed on that subject. However, 10 
were not aware of the poor compliance, or poor blood-pressure 
control of their own patients. Table 2 indicates that II of the 14 
physicians failed to mention paticrit education when they listed 
necessary interventions for the idc;»l manager^t^-nf of hypertf^nsion. 

The second questionnaire containing knowled^ items was directed 
toward thfi patients. The results had to be projected from the 
combined clinic data to a total group of 248 patients, because all the 
patients (52) were studied in one clinic, while only a random sample 
(100 of 196) were studied in the second dinic. From the sample 
group of 61 who saw the health, accountaii^t in the second clinic, a 
total of 120 patients were extrapolated and* added to the 28 patients 
who saw him in the first clinic, to obtain an extrapolated total of 148 



Table 2. Hypertension Questionnaire of Physicians on Value and Attitude 
Items 



Physicians* 
11 



Omitted the education of patients among aspects of ideal care for 
hypertension 

(^stioned the criteria for hypertension determined in a study by 
ihe Veterans Administration . 5 

Rejected clinical diagnostic standards for Evaluating the outcome of 
treatment • I 

Rejected the blood -pressure outcome project or the role of the 
health accountant • 1 



Total who were possibly deficient 



n 
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Table 3. Hypertension Questionnaire of Patients on Knowledge Items 





Patients* 


Unaware that they had hypertension 


23 


Unaware that hypertension is a serious risk to health 


14 


Unaware of thejform or dosage of the drug they were receiving to 




control hypertension 


30 


Unaware of the possible toxic effects of the drug they were 




receiving to control hypertension • 


120 


Unaware that the danger of hypertension is not related only to 




symptoms 


137 * 


Total with inadequte information** , 


140' 


•N-148 (exifrapolaicd) * * 


L 


* Counting each patient onlyNonce 





patients seen by the health accountant in both clinics. Similarly, the 
group of 100 remaining extraix)lated patients who saw only the 
physician Was obtained by adding the total numbej^ of patients in the 
first clinic (24) and the extrapolated sample of the second clinic (39 
extrapolated to 76). 

Table 3 illustrates the projected results from knowledge items in 
the patients' questionnaire. Of the extrapolated 148 patients who 
visited the health accountant and completed the questionnaire, 23 
were not aware that, they had hypertension; 14 were not aware that 
high blckxl pressure is a serious health risk; 30 were unaware of the 
form or dosage of the drug they were receiving to control hyperten- 
sion; 120 were unaware of the possible toxiq effects of that drug; and 
137 erroneously related the danger of hypeptension to svmptoms, not 
u^erstandiiig that their first symptom might be a stroflnr possibly 
death. T^ble 4 indicates that more than half these paticros were not 
obtaining adequate therapy for their ^high blood pressure. These 
findings^Constitute a set of correctable determinants of uncontrolled 
hyj[)ertension— the unacceptable health outcome t)figinally measii^d. 



Table 4. Hypertension Questionnaire of Patients on Behavior Items 

X . . Patients' 
Not takmg any medication for hypertension ' 39 
Taking medication for hypertension sporadically 25 
Taking medication for hypertension only if symptoms occur 16 

Total obtaining inadequate medication* .80 

* N-148 (cxirapobled) 
. ^ Counting each paiicni only once 
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During stage 4, an educational effort was Implemented by the 
health accountant. Each physician was given a specific educational 
prescription based on the inadequacies shown in the questionnaire. 
Depending on his answers, the physician was made aware of the 
compliance and blood pressure control rates of his own patients. 
Alternatively, he was given reprints of the Veterans' Administration 
cooperative studies about the efficacy of hypertension treatment' or 
of studies based on data from the surveys of the National Center for. 
Health Statistics which revealed the lack of relation of blood pressure 
levefs to symptoms. -Patients participated in a similar educational 
program based on those items that they answered incorrectly on the 
questionnaire. The health accountants had the patients return, first 
weekly and then every 6 weeks, for blood pressure checks and 
reeyaluations of their understanding of and compliance with the 
critical items included on the questionnaire. New health problems or 
lack of response to therapy resulted in an immediate referral to the 
physician^ 

In stage 5, a reassessment of health outcomes indicated that the 
educational effort had been an apparent success. Figure 6 indicates 
that the proportion of patients in the entire group whose blood 
pressure was out of contfol dropped from 36 percent in August 1973 
to 19 percent in August 1974. Although this figure was a significant 
improvement, it still was higher than the acceptable standard set by 
the clinic staff. Consequently, these patients were recycled to stage 3 
to identify other correctable factors. Here two independent groups 
were identified: (1) the 148 extrapolated patients who had responded 
to t^he health accountant's request to return to complete the questioh- 
nafre and .participate in the educational program based on its results; 
and (2) the 100 extrapolated patients who^refused to return to fill out 
the questionnaire, preferring to come in only for their regular 
appointments with their physicians; therefore^ the physicians were 
the main sources of their education. As figure 5 has shown, the 
differences in improvement in the two groups were startling. Those 
patients educated by the physician showed a drop from 39 percent 
uncontrolled to 29 percent; those educ^ed by the health accountant 
dropped from 34 percent uncontrolled to 13 percent. More than 
twice as many, patients who saw the health accountant had their 
hypertension brought under control. This difference in the rate of 
improvement was statistically significant at the p = 0.05 level. 

Utilization of Results 

The results of the overall health accounting research and develop- 
ment program have been applied by other- investigators and have 
mfiu^nced national quality assurance programs in the United States. 
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Figure 6, Uncontrolled blood pressure found in a 1-year followup of 248 
patienu panicipating in the study by HMO International [From Williamson 
el al. 1975, Reproduced with the permission of the BulUHn of the New York 
Academy of Medicine. Further reproduction prohibited without permission 
of coi>ynght holder.] 

Clement Brown was among the first to apply the basic health 
accounting strategy of cyclic integration of patient care assessment 
and .continuing medical education/ He developed an elaboration of 
this strategy/known as the "bi-cycle concept" of quality assurance' 
(Brown and Fleisher 1971). Hb version was implen^Mted in over 265 
hospitals ia the United States and became the basis df the Quality 
Assurance Procedure of the American Hospital ^Ikikjiation. Partly 
because of Brown's influence, the basic health ac|(!^u^rng approach 
was later reflected in PSRO legislation in the United S^tes. 

Ray Heifer was another early investigator who was^ influenced by 
the First health accounting studies. An article on I^^Wbrk at the 
University of Colorado Medical Center utilucei^ the ploflciency and 
efficiency health care quality indices dev;elqp^1byUhe ^uthdr in early 
assessment development studies relat^(l!to^ health ac^ai|ti^g (1967). 

Joseph Gonnella developed a tuntdiS? applicatiafip||>th4b health 
accounting outcome assessment kri^KJ^ ja^li^^^ the 
University of Illinois College of V^^(Micjb^ was 
reported in a subsequent article (Go^nelJiF^^ Hi& cpntribu- 

tion to quality assurance, focusing odM^wh- pi^^ 3tagitig/ repre- 
sents the subsequent growth of his 'if^^rest 4hV6^ 
methods. ' ^■'■^^'^ :>'^^i''-'^^ 

The JCAH was directly influence<f ^l^y heakhj^ its 
policy decision to adopt an outcome approach tCL.AuppIJ^ihent^^ 
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hospital accreditation procedures. Its Performance Evaluation Proce- 
dure, now implemented throughout the United States, is a variation 
of the health accounting outcome strategy. 

The HMO Act of 1973 was directly influenced by health account- 
ing. In this Icfgisiative bill, now signed intQ law, outcome-based quality 
assurance strategies are mandatory. 

Alvin Mushlin and Daniel Barr ( 1975) were directly influenced by 
implementation of health accounting in the prepaid group practice in 
Columbia, Maryland. They applied this outcome strategy in the 
development of a unique "Problem Status Index" for assessing 
outcomes of health problems in ambulatory patients. 

Robert Brook (1975) applied health accounting quality assessment 
design as part of his doctoral thesis research. The form and group 
judgment approach of health accounting outcoiie sUndards devel- 
opment directly influenced Brook's later research at Rand (Brook et 
al. 1976). ^ 

Thomas I niir was directly influenced by health accounting in 
development of his thesis work, "Improved Outcome in Hypertension 
after Physician Tutorials— A Controlled Trial," which was reported 
in a. subsequent article (1976). 

E>iscu5sion 

The methods and procedures of health accounting lhave been 
studied over a 14-year period. During the past 3 years, rigorous 
reliability and validity evaluation, has been applied to essential 
procedures with very encouraging results. 

The structured group judgment procedures for develo()ing priori- 
ties regarding topics for quality assurance study might be cited as a 
health accounting technique that has proved reliable and valid. Eight 
matched teams of representative staff in health care institutions 
..^ip^lependently identified areas of their own performance strengths 
^'^'^"^^ terms of patient health outcomes. The observed 
^<^n^<^r}cc of judgment could be expected less often than 1 in 
. 10,0€K)^;tf^<liily^^^ ajjreement were operating. The findings were 

rbased a<a^^^||^c^^^ observations of the procedure at work. 

Qutx)f IB j^ti^ltf to validate judgment, 17 have proved 

v^yali^l A^ of this effort has been the comparison 

"^^-^1^ previous nonstructured topic-selection 

•i^^^^^^ ^"^^^Pe^e^ced teams (group A), compared 'to 

by equally inexperienced teams (group 
?^ ^MWS^^^^^^^^^^ ^^^^^ (g**oup C). All teams were asked 
for ^^^^^ benefits not then being achieved^ 

m t|i|^^ practice but which they considered most achievable. 
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Group A, using nonstructured methods, identified only 2 percent of 
high-priority topics related to pediatrics. Utilizing structured proce- 
dures developed in health accounting, this proportion increased to 
10 percent in group B and 25 percent in group C. - 

Examples of the diversity of these pediatric topics developed by 
health accounting methods include the following: 

— ^Achievement of improved parent Compliance atid follow-up by 
family practitioners, pediatricians, and ENT specialists for recur- 
rent otitis media in children up to age 12; 

—Diagnosis, reporting, and follow-up on abused children under age 
9 by all primary health care personnel; 

— More specific diagnoses by physicians and school psychologists of 
learning disorders in school-age children; . 

— ^Treatment by- primary care physicians and mental health team of 
parenting problems. 

— Dental care to be arranged by administrators of all enrolled 
Medicaid patients age 4 to 16; 

— Improved provision of sex and contraception counseling by physi- 
cians, nurses, and health educators for adolescents age 10. to 18; 

— Detection by physicians and nurse practitioners of inadequate 
immunization status in patients age 2 months to 16 years; 

— Administrative arrangements to avoid exposure of well babies to 
acute illness in pediatric waiting rooms. , . 
. . >t 

Another important finding from an overall evaluation of health 
accounting procedures is that prospective follow-up -ministudies, as 
applied in this strategfy, have proved feasible and inexpensive. Of the 
first 56 projects completed, 51 involved assessment of health out- 
comes; 45 of these involved direct patient follow-up interviews from 
an aggregate total of 5,857 patients. Of this latter groiip, 4,714 
patients were successfully contacted and complete data were obtained, 
yielding an overall response and completion rate of 80.5 percent. 
This rate varied from 71.5 percent for poverty class patients, to 89.3 
percent of working class, and 91.6 percent for middle class patients 
sampled. 

Among the first 56 pi^ojects completed, 29 involved assessment of 
diagnostic outcomes based mainly on recorded medical record data. 
, Among 20 studies for \N(hich adequate denominator data were 
available, complete information was obtained for 14,020 of the 14,672 
patients sampled, yielding a data completion rate of 95.6 percent. 
This rate varied from 95.4 percent for poverty class patients to 93.4 
percent for working class, and 98.4 percent for middle class. 

These findings confirm the feasibility of conducting minisampling 
surveys of clinic and hospital patients for purposes of health care 
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outcome assessment. This capability is crucial for focusing quality 
assurance study on the critical problems, freeing the physician or 
researcher from the narrow constraints of studying only those 
problems identifiable in accessible data such as those found in patient 
charts. 

, The health accounting procedure has several- important advantages 
as a strategy to assess and improve health care: 

— Health accounting is a strategy that identifies deficient outcomes of 
health care and the deficient processes related to those outcomes. 
This strategy has the major advantage of considering all correcta- 
ble determinants of ijaaAequate healfih resOlts, not just those 
traditionally considereAs part of the medical care. 

— Health accounting encourages provider consideration of a broad 
range of problems, from diet ajid lifestyle to inadequate parenting. 
Encouraging certain children to learn better diet habits, to achieve 
swimming skills, or to clarify values regarding smoking and drugs 
may have far greater health benefits than most traditional immu- 
nization procedures. 

—By utilizing encounter, forms that are independent of medical 
records, health accounting can focus on topics having the greatest 
potential for effecting health improvement. In other words, this 
approach seeks data related to the most important problems, 
whereas present methods seek problems related to the most 
available data. 

— Health accounting stresses quality assurance clinic staff participa- 
tion in an educational planning process that requires a series of 
cost-effectiveness decisions at key points in every project. In this 
process the staff must examine its assumptions regarding: 

—Prevalence of the health problem in their own patient popula- 
tion/ 

— Scientific evidence of efficacy of health care interventions. 
— Level of health or economic outcome deficiency in their 
present practice. 

— Probability of achieving significant. improvement if the above 

deficiencies are verified. 
— Probability that the improvement achieved will be worth the 
total quality assurance effort and costs involved. 
-^Health accounting requires active personal involvement of the staff 
in a rewarcTmg procesfs that entails a focus on the validity of their 
present knowledge of health science information, as well as clarifi- 
cation -of their values and their assumptions about those patient or 
consumer values that are encompassed in establishing quality 
assurance priorities and outcome standards. 
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On the other hand, health accounting entails several disadvantages:" 

—The concepts and procedures required by health accounting, such 
as epidemiology, efficacy, and sampling and survey methods, are 
often foreign to health care personnel, and must therefore be 
learned and understood. 

-^Developing skills in these procedures requires contact with trained 
personnel who are in ^hort supply at present. In the interim it may 
be necessary to hire a coordinator who is familiar with research 
and evaluation methods. 

— The act of exposing cherished health management assumptions 
held by staff members to open group critique can be an emotionally 
threatening experience'. Assumptions regarding the validity of 
diagnostic methods or efficacy of the therapeutic modalities often 
prove to be ill founded. 

—Tracking down unavailable patients that fall in study samples can 
be a frustrating as well as a challenging experience. 

-^Involvement with this method requires considerable internal moti- 
vation of the participating staff. This requirement is often difficult 
to meet when the staff is concerned more with productivity than 
• with the improvement of health benefit. 

In. conclusion, experience to date indicates that there are some 
financiai and staff costs involved in implementflig the health account- 
ing strategy. However, these costs do not seem inordinate and are 
probably much less than those for alternative systems. On the other 
hand, the potential health impact of this strategy seems to be 
significantly greater than has been documented by other systems. 
The educational rewards and enhanced personal staff involvement 
,alone s«em. worth the total effort. Finally, the extensive and favorable 
reliability and validity testing that this method has undergone makes 
it unique and worthy.of important consideration. 
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DISCUSSION 



Milton L Roemerand Gabriel Velazquez 

The combination of a method of quality assessment with strategies 
for promoting health care improvements, based on the findings of 
Dr. Williamson's "health accounting/* is especially noteworthy. It 
appears that the personal involvement of a broad range of doctors and 
other members of the health care team is the crucial feature of the 
process.. 

It was noted that topics selected for evaluation were predominantly 
practices for which there was a broad consensus on efficacy— such as 
drug therapy of hypertension or cerUin immunizations. Thus, the 
task is only to determine if the specific service is being effectively 
delivered. 

In the early stages of development of the health accounting 
strategy. Dr. VJfilliamson and his colleagues took the initfative in 
approaching selected group practices. As the work became better 
known, various health care organizations came to him for evaluation. 
In all cases, the final definition of problems to be investigated was 
^made by the personnel in the cognizant organization, not by the 
•outside" evaluators. This seems to assure that corrective actions will 
be taken in response to the findings. Dr. Williamson stresses that the 
findings, however, are not overextended beyond the specific problem 
investigated. ' . - 

The management of certain disease entities has nearly always been 
the topic initially chosen for evaluation in any one health care 
organization. With time and experience, more attention tends to be 
directed to issues in prevention and in the organization or administra- . 
tion of health services. 

A key problem has been the training of competent "health 
accountants.** These persons must be carefully chosen and skillful in 
their interpersonal relationships. The training of health accountants 
has now been standardized, but it involves a cost— not to mention the 
cost of the operation of the evaluation program over time. 

So far. the method has only been applied in organized medical 
care settings, mainly group practices. It is hoped to apply it eventually 
also to solo medical practices, perhaps by assembling "consortia" of 
private practitioners in an area. Since the method is prospective! . 
rather than retrospective, it should be feasible to collect the necessary 
information (through records), once a decision has been made to 
undertake the evaluation. 

The health accounting strategy for assessing quality through 
outcome measures has been tested recently in Holland. It might be - 
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more difficult to apply in Latin America, but it would seem to be 
tirorth tfying in a program like that of PRI MOPS in Cali/l:>>lombia, 
where a strong, technically competent sufT is assembled. ^ 



^•1 / 



. '16. . 



MIRROR OR WINDOW: THE INTERFACE OF HEALTH 
» .SERVICES EVALUATK^^N AND PROVIDER EDUCATION 

• ; JohnL.vimon ^ 

In a very important sense. /provid^/ education is the interface 
^ between the fvattiation oC-Kpalth' services and medical practice. Its 
responsibility is to use evaluative data' to keep its currifrulum "healthy" 
in the hoije that tjie m«iUcaf practice of its graduates will be 
improved. The extent^ which this interf^can be characterized as 
a ihirror or as, a window may indicate how gqod a fit there will be 
berween what .we are'^raining our providers to do, and what our 
providers will need to be able to d6. • 

If we were preparing to dress ^appropriately for the weather 
outside, we would be m'ore likely to *look through a window than a 
mirror. -If the sun were shining, some of'*ls would dress for warm 
weather. Hbwever. we might telephone* the weather service for a 
temperature reading, because the sun can shine in 32° or 52° weather 
Some of us. might still not venture out for a long trip, without some 
weather forecast. Will we need an umbrella 4 hours from now? After 
collecting all these data, we might dress appropriately and tfum stand 
m (front of a 'mirror. Do we look presentable or ridiculous? Can we 
modify our clothing combination, or is this choice tTie best we can 
make? In summary, both the wihdow and mirror are important tools 
if we , are to dre^ appropriately- for weathering the environment. 

TRis simple analogy:; tati^ also be applied to evaluation of health 
services. Health serj^iies evaluation holds the potential to be. a useful 
jWindow to the environment into which we s^d our students, as well 
as a mirror reflecting how well we have prepared them for it. While' *" 
such evaluation may yield, imperfect and imprecise information, it can 
reflect to some extentjwhether our graduates aVe presentable or 
incongruous in terms of tljrir training; or whether we need to modify 
the clothing combination or curriculum." , * '^^ipw 

If my undierstanding. is correct, there are two major issues concern- 
ing the delivery of^ child health serffces in the United States, today. 
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The first is identified by using evaluative research as a window, 
Haggerty and others (1975) have looked through the window and 
described a **new morbidity/* Thi» fresh and perceptive look at the 
child health environment indicates the need for new knowledge, and 
therefore must impact on the education of child health providers. 

The second major issue is identified by using evaluative research as 
a mirror. Richmond (1977) and Ressner (1974) have made us face 
the mirror to discover that even our existing knowledge is not being 
adequately applied. This information, too, must have an imp>act on 
the education of child health providers. 

In essence, I am suggesting^^ the rather simple notion that the 
educational objectives of a curriculum for health providers ought to 
be derived from the professional tasks and responsibilities these 
future providers will have in their professional settings. Moreover, 
this use. of actual experiences for planning educational objectives 
m list be dynamic. The results from health service evaluation must be 
continually assessed^ and translated into educationally useful data. 
The fact that curricula in health science schools usually represent a 
measurement of the relative strengths of department chairmen, 
expressed in terms of hours, suggests that this *'simple" notion is not 
quite so simple to implement, 

BACKGROUND 

Our project was an embryonic attempt to use several types of 
research data as a source for a minicurriculum based within a Health 
Maintenance Organization (HMO), The project took place at George- 
town University from June 1975 through August 1976, At that time, 
I was a^fac^ty member in the Department of Community Medicine 
and International Health,* 

The Georgetown University School of Medicine is a private, urban 
medicaj school located in Washington, D,C, There are 205 students 
enrolled in each class; Except for a 2-to-3-week' stint in a hospital 
^outpatient department or physician's office during medical and 
pediatric clerkships, no primary care experiences are required. 

The Gc^orgetown University Community Health Plan (GUCHP) is a 
prepaid, group-practice-type HMO whicih^opened fpr services in late 
1972, Originally sponsored by the Departme,nt of Community Medi- 
cine ^nd International Health, today it is an independent corporation 
-serving an economic cross section of the Washington metroiX)litan 
community. At the time of the project, there were over 10,000 
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cnroUccs and an additional 6,000 **active" fee-for-scrvice patients who 
were served by the HMO s three primary care centers: one in densely 
populated northeast Washington; a second in the new town of 
Reston, Virginia; and a third in Kensington, Maryland, Community 
hospitals were utilized by each primary care center, and Georgetown 
University Hospital was utilized as a tertiary-€are- referral center. 

Our goal was to use GUGHP iis a setting for teaching aspects of 
primary care, to undergraduate medical and nursing students. The 
message I would like to cpnvey, however, deals with the ^process we ^ 
used to plan the minicurriculum, rather than the specific setting. We 
began by asking'the following kinds of questions. If **child health 
provider" is substituted for "primary care physician," the following 
questions will still be the relevant ones to ask when planning a 
purposive educational experience: 

—What is a primary care physician? 

—What constitutes the content of primary, care? v 

— What responsibilities or competencies are, or should be, required 

of the primary care physician? 
— What .effect does professional setting have, be it HMO, solo 
practice, partnership, etc., on the role of the primary care physi- 
cian? 

The answers to these questions provided the educational objectives 
for the curriculum. Obviously, it would have taken several years and 
2l;/major "study to answef these questions with precision. We had 
neither Ihe reiiourcts nor the luxury of time to carry out such a 
study, so we approximated one. We did some evaluation of our own, 
as well as usmg research reported in the literature. 

In an effort to look through the appropriate windows, we thought 
it w^s very important to understand! 

« - . 

—The perspective of the learner, i.e., medical and nursing students; 
— The perspective of the potential preceptor, i.e., the primary care 
' providers of the HMO; - 
— The perspective of the consumer, i.e.. the HMO enrpllee. 

THE PERSPECTIVE OF THE LEARNER 

We hired three sophomore medical students and three seniop 
nursing students for a summer. They formed three interdisciplinary 
teams, each consisting i>f one nursing and one medrral student. 
Under faculty guidance, these student teams searched out and read 
literature on issues in primary care. They studied the contenrof 
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primary care practice ana the roles of the primary care physician and 
nurse pracuuoners in a health team. They also read about successful 
and unsuccessful curricula attempted elsewhere for training under- 
graduates in primary care. The learning that occurred during this 
phase prepared the student teams for the next phase. 

THE PERSPECTIVE OF THE HJTURE mCEPTOR 

The studenu* next task was to discern certain aspects of primary 
care in our HMO setting. Because of the students* level of clinical 
and technical expertise, they concentrated or the more generic tasks 
and responsibilities of primary care physicians and nurses in GUCHP. 

Each interdisciplinary team was assigned to one of GUCHFs three 
primary care centers. The teams first developed observation guides 
and spent I week observing physiciai>-nurse. physiciai>-patient. and 
nurs<y-patient interactions. Then, based on their literature reading 
and their observation results, the students developed a staff interview 
questionnaire. Each team interviewed the primary care physicians 
and primary nurse practitioners in their centers" individually. Since 
the pracuuoners were our students* potential preceptors, the faculty 
thought It was crucial to understand the practitioners* perceptions of 
primary care, practicing within an HMO. and working on a health 
care team. In addition, their views as to the adequacy of their 
previous training in preparing them for their current roles wtire also 
sought. Some of the questions asked of staff were: 

—What do you find satisfying about working in primary care? 
—What do you find unsatisfying about working- in primary care' 
—What kinds of things have yqu encountered in primary care 

practice that your education didn't prepare you for? 
—What do you feel are the most important factors which influence 

your role relationship with the physician/nursp? 

How can the answers to these questions help in planning a 
curnculumPAn example is our finding that many physicians bllieved 
they had not been sufficiently prepared for their supervisory respon- 
sibilities. The addition of supervising students on top of other 
su^rvisory tasks augmented their need for such skills. Therefore 
ontf of our planning needs became the development of a preceptor 
preparation course. Virtually all full-time primary practitioners vol- 
unteered to participate on their own time in a 3-session,- 6-hour 
course. A major component of the course included real and simulated 
trainmg in a teaching technique called "clinical supervision." The 
curriculum, the resources, and some of the exercises used in this 
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I course are currently being published for use in other preparation 
courses for clinical preceptors (Simon!, in press). 

• ■ ' . [ ■ 

THE PERSPECTIVE OF THE CONSUMER 

* Each of the GUCHP primary c^re centers ha^ a Consumer 
Advisory Board, The faculty initially discussed the project with these 
boards at the HMO centers. Both faculty and students^thought it was 
essential to learn wh^l patienti (iiT this dise, HMO erirollees) expected 
from-the physician. It was felt that \ patient expectations of^TGg 
physician and methods for dealing with such expectations should 
definitely be included in a training progtaim for primary care. * 

V Therefore, the students developed a questionnaire and interviewed 
enrollees both brfore and after their enqoiinters with the physician or 
nurse pracutiqrier. .Examples of, questions: asked of patients Z^^/or^ 
their enCi(>unter included:; * • " ' 

— What db^ybu' Want to happeiuiSiiring ypur visit today? . 
— Do yon fia^je any questio'dsron yoiir^ mind you want to ask the 
doctor or nurse? 'i; ' ^ • . ; • 

• ■ t>'' ■ < ' 
Examples of questions aslci&d ?^f|^J)ai|ients ^^^^^ 
included: * - ^ i * 'i'^. r / ' 

—Did you ask all of your questions? 
— Did you get. all of your questions answered? ^ 
— What things were satisfying about your visit with the doctor/nurse? 
— What things were not satisfying about your visit with the doctor/ 
nurse? 

In this case, we were not sp>ecifically interested in whether or not 
the patient was satisfied. Rather/we were concerned with what 
incidents were considered critical by the patient in determining 
whether the visit had been satisfying or ulisatisfying. 



PRIMARY CARE COURSES 

Patient responses to those questions were being summarized when 
I left the project. A preliminary look at these data, however, has 
already generated ideas for the following courses, which were being 
planned. ^ . j 
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A Course in Developing a "Good Chairside Manner^', / i v : / ' 

During an interview with a student, one patient responded thai the 
factor which satisfied her the most about her visit witb' the ^lpctof, W^^^^ 
his "good chairside manner." Evidently, the bedsid<^ njarwie^ 
inpatient care has a corollary in the chairside manner pranibu^^^^ 
care. Other patients' responses helped to . defme a ^gbi^ -chaiTsi^^^^^ 
manner; based on this information, a course is tbvb^^^devfeloj^ 
first-year students. It will concentrate on the-paiienf-rphysicia^^^^^^^ 
tionship and invofVe interviewing techniques, tiuch ^^s pptl^W^^^ 
patient at ea.se and managing encounter time M;)"/thut * padem* and' 
physician do not feel rushed. It will also include techtiui^esvfQ^ 
answering patient questions. ; f ' ; . , ' 

A Course in Telephone Medicine . / > ! ' ' / " ' 

One outcome of our evaluation of the 'j3i^6fAsioriat..t^ 
responsibilities of primary care practitioners r^aled to. l^^^ 
encounters. We found that a growing propoHibtv of p^ 
being delivered over the telephone. Physicikps-'in ;t^^^^^^ States 
are spending 1 to 2 hbm s pec day C()nsuldrig With Pa^^^ phone. 
Child health providers are perhaps t he' heaviest utilizerrof ^efe^^^ 
medicme. In most instaijces. tliese providersVWvaiiablyCnW 'only: have 
to make clinical judgments without ihe p'ane'ritVs:^bfeihg vislb^ 
but also must elicit xriticaf information in an .effg^ a* 
third party— usually a parent or gtiardian. ; '' '^^^ 

Alttiopgh pediatric^cl^^rkS or< house offureii^s .probably participate in 
^telephone duty or nigh't call, participatlpn/te^ not eqa<il= education 
(Ott et al. 1974). The skills ^and knowle^ige.^^ 

^•treat by telephoned neec^ to be documi^hjLed and analyzed: It is a 
facult/s responsibility -to systematically plan; b^^^^^^ 
experience' through which a learner^k^cotnijetent^e 
medicine can be gaihed and assessed. • 

Electives in.Pi^mai>;CafeJeam Practice^ ■ ; ^ -^v;^ 

One outcome of the medicaUnursing-^tqderil^eanl. expcjriente was 
data fof; planning ^ sequence ot* two primary care leai^ The 
firs? elective would be. targeted jQiatJy^fpf ■s<i.phdmd^^^^ mtdical 
students and Junior nursing studejots. this cour^^^ the 
students into MD^RN student dyads or teiin)s-an^ on 
developing •^skills required for ■ effective /riurse! li.ractitioner-primary 
care physician collaborative practice: kole negotiaxion practi- 
cal exploration of each pther^ prpf^s^ioiia) skjl^ be a course 
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goal, and methods of simulating team practice would be heavily 
utilized. The major goal of this course would be to train student 
physician-nurse teams just to a ipvel where they could treat patients 
with minirtial disruption to norn^l HMO operations. 

The second experience, involving senior medical and senior nurs- 
ing students, was piloted last September, This course is a 6-Week 
clerkship in the Health Plan. The medical students and nursing 
students again constituted primary care teamsjr precepted by mentor 
primary care teams within the HMO. 

At this point, let me summarize what we have accomplished. 
Instead of meeting as a group of faculty to plan a series of courses in 
primary care, we first went to ^ < • 

— the practitioners of primary care 

— the recipients pf primary care 

—-the potential stt*dents of primary care 

to help answer the questions originally stated. We attempted to utilize 
existing and newly generated evalp(UiYe'rps;ults a basis for planning 
ed uca t io na 1 o bjec t i ves . ' ^ y . > "if^: / . .V\> ' 

I ACHIEVEMENTS AND PROBLEMS 

Achievements 

—The types of questions. we. were asking are slill being asked and 

/.answered , in the HMO. The providers themselves have "taken off 

; ^^h' some of the data produced .by the, students to imprpve their' 
own performance. ' * - .^n - - } 

— ^The students, while evaluating' ottly /dxf aspect of health practice, 
were treated to an intense learning experience, the product of 
which will be educational objectives for future students. 

— ^The positive attitudes of the providers toward evaluation served as 
a good role-modeling experience for the students.^ 

—The process of learning about professional roles in this setting has 
generated A Medical Students' Guide to Health Maintenance Organiza- 
tions (Simon et al. in press). This guide is targeted for classroom 
use, Essentially written by medical students for medi^Cal students, it 
is currently being revised to include the HMO Amendments of 

^ 1976 , . V , 

Problems ■ 

# ' ■ ' 

— ^The project did not last long enough to institutionalize the process 
within the department.. 



EVALUATION OF CHILD HEALTH SERVICES 



—Many of the educational objectives to be derived from ouV data arc 
probably more relevant to the teaching programs of other depart- 
ments within the medical school, wher^ the mission for training 
clinicians is more legitimized traditionally. 

* . CONCLUSION . 

The approach for curriculum developmjpnt described above repre- 
sents one way to utilize the interface of r.i;t^rch and medical practice 
as a window and rhirror for provider edy^ation. A- similar process has 
been applied on a broader scale by others. In his recently published 
book, An Inventory for Primary Health Care Practice, Dr. Archie Golden 
(1976, p. 7) writes: ^ , . 

^ .Jhcre is little evidenc^ today that curricula for ali levels of health 
* 'woj|-kers are based on the real world of the performance of heahh . , 
personnel in the\field. There is a need to relate the disparate areas 
of education for health careers and the delivery of services. 

I have attempted to make a case for utilizing data jfrom health 
services evaluation to develop a set of educational target for student 
learning; .There sho^^.ld be better examples of how best to do this: 
Perhaps you will help to find them. 
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EVALUATION AS A TOOL FOR COMMUNITY 
EDUCATION* 

Rolando Merino, Samuel J. Bosch, Ruth Zambrana 

Introduction 

During the last dccadfe in the United States,' consumers and, 
community groups have Expressed clear demand fpr participation in' 
health institutions and, more importantly, in. the decisionmaking 
processes of those health organizations that affect, their liv^s-(Gartner 
>nd Riessraani 1974, Metsch and Veney 1976). This demand^as not 
; -led to participation in established health or.ganizations, but rjfther tO; 
the development of new, community-based organizational. modalit^iesj- 
such as neighborhood health centers, most of which were launched- 
by the Office of Economic Opportunity (OEO) and later sup|il^ 
mented with centers funded by the Department of Health, Education, 
and Welfare ''(DHEW). Neighborhood health centers (NHCs) were 
intended to exert a direct influence on the way medical care was 
provided to persons in low-income areas. NHCs were to provide a 
broad range of services, including outreach services by indigenous 
workers and legal services not usually within the purview of a health 
facUity (Zwick' 1972, Klarmah 1974). They were also meant to' assume, 
with consumer participation, a measure of responsibility for the 
continuous health care of the populations that they register (OEO 
1968). Evaluation of established health orgar^izations has been man- 
dated by several Federal programs, thereby presenting difficulties in 
the selection .of measurable criteria, as well as the implementation 
and assessment of programs. Community-baiqd health centers do not 
have' the advantages of the established medical community. For this 
very reason, it is vital that community organizations be capable of 
effectively evaluating their efforts to provide heafth services in 
previously iinderserved and needy areas. 

Evaluation data were used as the prjncipartool in an educational" 
program planned and developed by a community organization in 
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partnership with a school of medicine. Thjs paper presents the ' 
preliminary findings of that program. The program was implemented;!^ 
lo mcrease the health care knowledge and decisionmaking skills of 
the consumer governing board irt a recently established NHC that 
$erves children and their families ih an underprivileired area of Ne " 
York City. ' ■ i 




Background ^ " ' 

In East Harlem, 48 percent of thq population is of Hispanic' origin - '^ 
Gohnson et al. 1972). On July, 1, 1975, the East Harlem Tehant^^ * 
Council (EHTC), a Puertq. Rican^ community-based organization, took 
over a New York City Health' Department 'thiid health station aqd « \ 
converted it into an NHC with 'aij' HEW 314=^E ^rant. The center now . 
serves the children previously servjed by ;the health station, their 
families, and a growing; number of new families in the area. The / ' 
EHTC received technipl assistance in thfe planning and development 
of. the center from the^Dcpartment of Community Medicine (DCM) 
of. the Mount Sinai School of Medicine (iosch and Deiischle 1977), 
Since then, the center ;has operated under^he supervision of the 
EHTC'sJay board with the parti<;ipaUon* of consumers (registered 
patients) on the board. The DCM'maint^ns its role in the evaluation 
of the center's program. On April 1, 19^6, the EHTC and the DCM 
^5^^^^^ ?9 undertake a joint educational program to enhance the 
EHTC boards capacity to govern the center. 
The EHw board is composed of 18 persons (from various 
.jdiscipli^ierandVAvith different educational backgrounds) wjio are not 
health care providers! The m^ority are East Harlem residents and ' 
consun^ers of servicei in the health center. The board is the policy ' 
maker Yor the NHC. Its policies at-e communicated through the 
project director, v/tio is responsible for their execution. Board 
lead<*rship belongs to its Executive Committee*, composed of a 
chairperson, a vice president, a treasurer, a secretary, "and two ^ 
members-at-large. Ther^ are two standing committees— Membership , 
and Personnel — and ad hoc committees are formed aV needetf. 
Elections to allycommittees are held annually or as vacancies occur. ! 

Th^ Need for An Edjicational Program 

In the original legislativie mandate for NHCs, participation was 
required in those federally funded programs where policymaking 
responsibilities were specified for the consumers in planning, man-* 
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agement, and evaluation activities (OEO 1968), The requirement is 
based on the premise that consumers are more sensitive to- and 
respond belter to community health nepds. Therefore; if they could 
participate in policymaking, the goals of a'progrum would be better 
met and the NHCs would become more readily accepted:^ in the. 
communities they serve. Experience has demonstrated thai participa-- 
lion can occur if consumers and community organizations take the 
necessary steps to acquire the knowledge and skills that are pertinent 
lo .their particular area of decisiohmaking, ^ 

During the planning of the center's first-year operational phase, 
1975-76, the active participation of community members through the 
\ EHTC board produced a health care plan tailored to meet th'e health 
. and medical care needs of the East Harlem community. Nevertheless. 
, during the first year of services there w^s little participation by the 
, EHTC board in the management— in the definition of overall'policies 
>ai)d .in the periodic evaluation of the program. During this period, 
the board realized that it was gradually losing control of the program 
and recognized t;he need for more expertise in health care policy in 
order to articulajfe the health needs of the community more effec- 
tively. At the same time, the authors of this paper, functioning as the 
DCM planBiipg and evaluation team for the center, also recognized 
that the board ifnembers' inadequate understanding of the center's 
se^l^ce goals was encouraging political interests to take priority over 
program objectives. We also observed that the absence of basic health 
care knowledge on the part of the boar^ was permitting the center^s 
administration to decide on matters thift were rightfully under the 
jurisdiction of the consumer board. In other words, our observations 
confirmed that the jgoverning board's power over the center was 
gradually being shifted from the consumers to the providers of care. 
An analysis of our observations enabled us to conclude that this shift, 
was occurring not as a result of a power conflict between the board 
and the administration, but primarily because of a deficiency ia the 
technical capability of the board to define health care policies. Such 
cooptatipn of the community organizations' decisionmaking power by 
administrators is frequently encountered. Since a power conflict' Was 
not the issuie here, an explanation was sought in other arecfs. 

Several discussions with the project director identified the need' for 
more active consumer participation in the governance of the cenjter. 
It was 4lso hypothesized that active board participation could best be 
attained if two basic criteria were' followed: (1) The board should 
learn how to read and analyzp the basic utilization and cost data used 
in the periodic eviiluUtion of the center; . and (2) the board should 
learn how to Understand basic effectiveness and efficiency measures, 
so that its decisions would be more relevant to community needs. 
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The Development of the Educational Program ; ^ 

The planning and development of the curriculum were carried out 
by four DCM health planners^ who represented differem disciplines, 
and the NHC project director. As trainees in the program, the 
members of the EHTC board actively participated in 'the definition 
of^ the educational objectives, selection of the educational methods, 
and developnient of evaluation procedures, ^ <t 

The educational objectives of the program were defined through 

—Analysis of the results of a questionnaire administered to the board 
membera to identify their perceptibhis of learning needs; 

— Obsej^^tion^ by a bilingual Puerto* Ricah sociologist of three 
montfily board meetings that preceded the program; 

— Direct participation of the center's project director in, the planning 
' process. . / ' - . 

The ieducational objectives were'defined to ensure that boai'd 
memhiers vyould have acquired Certain skills and knowledge at the 
end of 1 year. The ma?eimiim 15 hoiirs available for formal training 
sessions was also taken into consideration. ./ V 

It was agreed that at the end of the training program the board 
members should have the skills to; \ 

— Read monthly utilization repor^^s.ci itically. These reports included 

data on th^ efficiency and effectiveness of the health center. 
— Recognize utilization and productivity trends. 

T— Utilize data. reports to decicle among alternative recommendations 

made by the center's direttor. 
— Critically analyze the budget jn relation to the health needs of the 

community and the health ^renter's objectives; analyze data related 

to the effectiveness and efficiency of the health center's operation 

and to its financial sources of income. 

It was also agreed that at the end of the same period, member^ of 
the board should have acquired: knowledge of the terminology and 
basic concepts involved in (he following specific topics: 

—Community health needs, demand, and utilftation of health serv- 
• ices; 

— Utilization and productivity trends and relateq^'fiictors; 
— Program budgeting; 

— Ongoing evaluation of the center's performanceV^^ith emphasis on 
program efficiency and effectiveness; and ». ^ 
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—Definition of health policies and intermediate and long 'term 
planning. j \ - 

The training methods were defined on the basis of suggestions 
made by the chairperson and executive commirtee members of the 
board. It was decided that tfie mdst appropriate teaching procedures 
w.ould be ^ ^ : ' ' 

—Prescjitation of educational materials in an increhiental ftfchion, 

through group discussion with the trainers andurainees; ' ' 

— Focusing, discussions on analysis of (he centers actual evaluative 

data; . ' . ' / • 

.—Presentation of these data by the director of the health center to - 

the board members; 
—Explanation of selected topics for that sessipn using actual data. 

following th^project\director\s presentation; 
—Encouragement of di^ussionibetween the boar^jnembers and the 

trainers based on questioh^ and answers. l 

The-main educational resour<;;es were the DoM hedilth planners, who 
assumed the responsibility of educators throughout the program, jf^^ ^ 

The summative procedure utilized in the evaluation of this pro- 
gram's results was the administration of three multiple-f hoice ques- 
tionnaires to board members in order to measure their knowledge 
gain. The final questionn^aire will be administered upon completion 
of the 12-month training period. Results of thir procedure are 
therefore not incorporated in th.is paper. ♦ 

The main formative proce<lure was the field observation of **a , 
group at work" in order to ^evaluate process, particularly in 'relation 
to participation in decisionmaking. The field observations were made 
by the same sociolqgist who participated in the identification of 
'educationa^ objectives. In this particular training program, the sociol- 
ogist attended (as an observer) all board meetings and executive 
sessions, in addition to the curriculum development meeting^, to 
collect detailed descriptive data and^Jto analyze them in terms of the 
program's objecdves. The major thrust of the analysis was to identify , 
the -crucial variables of group process— such, as intragroup conflict 
and hidden agendas of individuals — which have an impact on the 
decisionmaking capability of the board. The principal* methods^u^edf 
Were qualitative analysis of data bas6d on the following criteria: 
increased level of participation;^ use of appropriate terminology ; better 
understanding of the relationship between incpme and expenses; and*^ 
application and use of information presented in the training sessions 
in ordjr to make policy de<S)sions. The predetermined 'categories 
were developed in consultation with education experts.'^ ' ' 
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Preliijilhary Findings v 

In general; the EHTC board menfibers are 'demonstrating a 
favorable attitude tov«ird. the .educational, graining process. The 
following preliminary results indicate some, positive outcomes of the 
training program: * , 

— increased participation in dLsciusions. An increasing nuniber'of board 
members have gradually begun t(f participate in the different 
sessions. Further, the ^formulation of^ pertinent questions, opinions, 
and suggestions in^. relation to tKt management of th^ health center 
has become njore sophisticated. 9, 

-^ore appropriate use of terminoio^ and concepts. Board members have 
also made progress iri using appropriate termin^lojfy and concepts 
related to the operation and evaluaiion^f tRe performaifce of the 
health center; Of utmost invportance has been their ability to use 
and interpret the^ measures of effectiveness, for example, compari- 
son between attained and planned objectijives, activities, and re*- 
sources' This ability has in turn enabled them to recognize th«* 
concept of efficiency the relationship between ongoing activities, 
cost per activity and administrative costs, and^the cost per unit 
distribution in operations. Th<f .relationsl^ip between the unit cost 
of a specific activity and outcome in tetms of heali(j has also been 
explored by board mernl)ers. 

— Better understanding of relationship between income and expenses. The 
board members have acquired an I'inderstanding of the notion of 
limited resources for unlimited health needs. During the last^wo 
session.s, they have initiated in-depth discussions of priorities in the 
forinulation of objectives for the health center. There have also 
been several discussions on the identification rtf activities which 

% would enable the center to operate more effectively: the cost of 
pediatric care as compared to adult care and the impact -of both on^ 
the health status of the population, as well as the cost of preventive 
care as opposed to diagnostic care and treatment. The need to 
establish health priorities has been recognized as a fundamental 
planning concept. 

— Request for a standardized information system for alpfihgrams operated hy 

\the council. An impoi/int outcome of training has been the 
♦interpretation and usVof monthly statistical data pertaining to the- 
health center's fupc/ons. The board has suggested the possibility 
of .standardizing a periodic information system for all the programs 
presently operated b>^ the institution. This would involve defining 
indicators that would demonstrate efficiency, effectiveness, and 
quality of services delivered by the different programs under the 
auspices of the KH I C:, ' 
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--inclusion of topics: not originally de/med in the curriculum. There fias 

* been a persisten^t demand fr6rft the board "^memljprs to engage in 

: detailed discussions F%atding' planning, momtoring, and evalua- 
tion of the health center pr^jram. The assessment ofahe quality of 
Oare, for example, has beejn identified as an area \^h should be 
addressed in a^uppIemenlSS trafling sKsioh. 

-^j^nsion of the designate^kf^m^or the training sessions, rffiitiafiy it was 
detided, at the suggestion of t>ie bo^^tl ^pembers, that each sesHon 
would last no longer than 1 hour. However, the se^sion^ 4iave 

generally been prolonged to^.lV*-2 hoilP^because of th^ interest 
generated *by the materials "presented. Evei]^^, it has become 
necessary to set limits on the.q^ue^i'ons and anJv^ raised by board 
members, especially when the ^questions wjr^ not related to the 
material Jjeing covered in a particular session. T^fe limitation was 
sel so that the trainers could systematiijaily develop the cUrrieulum 
in its totality. ^ . / 

—Request ^ additional training session^^' The boards chairperson has 
made^ a formaU'request . for^ additional trainirtg sessions, ^idte 
sessions would be provide.^ for .those ^members whq^.desire to 

^further^expand jheir knq^vledge in the areas alreacfy presented in 
the regular se^sion^ antd for the new' members .who joinecf the 
board aftej- the initiation of the tfaiifing program. * * 
^—Improvement of the attitSiMttlif the boar^meinbers *towtird the DCMt^ 
Anotner important •buttWie has-been an improvement in the 
relatibnship between the ErfTC an^t the DCl^Interaction during 
the training session* has h'elpec! to establish better communication 
between theni. ^ * ttfl 

«hally,^the board .members^^Kided to review in Fe^uary, at a 
specially designated meeting, a final dr^ft of the 199^-^8 NHC 
^ proposal which was submitted to DHEWifl N^rch'^977. Th^ review 
^rant proposal by board, members is,i standa^rd pfetedure 
which is consited part of th^ii^ responsibility as^nand^ed by the 
legislation for NPhCs. The board mq^ers in that meifeng were able * 
to r^vieW'the goals and objective^of the programti^ecifically^ term?^ 
of personnel and activities, review and re^fine the^ctiviti^Fn^ces 
sary to achieve the goaTs^and objectives, and aftalyze appro^e't^e 
budget as proposed by the project director. 

Discussion and^oncWisione *' 

Tjie . major problems in^the development of this mo^t were 



encountered in the identification and aprplication or evalMdhJbn 
instruments that could measure with "precision the* process, the 
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acquisilibn of knowle^e, and ihie lear#!ing of decisionmaking skills. 
As expected, the heterogeneity of the board in terms of education/ 
and professional* background made this task difficulty One of the first 
problems was the board's initial reluctance to grant the sociologist 
permission to attend meetings as an observer. In essence, the 
members felt uncomfortable with the fact that an "outsider" would be 
privy to aU their deliberations. Several discuisions were held in order 
JA.^ clarify the^iole of.the sociologist, particularly in terrns of confiden- 
tiality. Ultimately, a letter of agreement w3s wjitten to ejisui^ 
confidentiality and the right of the board to request that the observer 
leave the board meetings at any time the members considered it 
necessary. . ' 

The administration of the multipjp-choice questionnaire was not a 
major problem to overcome; nevertheless, like any traditioiial^ethod 
of evaluation, it raised questions by the project director aAd members 
of the lx>ar:d regarding the advisability of doing research "on them.'\, 

, The Ijjoard.iriembei ^i, all very busy persons iit their usual occupations, 
havej)nly a limited amount of time to (levote to nonfirofit activities, 
A tirrte-consiiming procedure was out of the question. An agreement 
was reached whereby a short questionnaire would" be administered 
during the meetings. The admission of new members to the board 

-» midway in the program complicated measurement even further. 
Consequently, i\}v impact of our^intervention cannot be measured as 
a cause-effect it^teract ion. 

Our preliminary findings merely indicate that a cl\^nge has 
occurred as a result of ihe program. More importantly, the introduc- 
tion of the progranx has served as a-ratalyst for change and has 
accelerated a process whose effect is not immediately meSs?irable, 

/ Without doubt, one of thp principal outcomes of this progranvhas 
be'c^ the improvement of interinstitutional, as well as interperso\ial, 

V relations. Th^. process at this point appears irreversible, mainly 
because of the mutual cognitive and affective learning that is taking 
place among trainees and trainers. P*urther, the interaction which 
facilitates that learning Ts inHuencing attitudes and strengthening 
tVisf and rapport between the fKvo*|^roups and the institutions they 
represeift. ,^ > ^ 

, Probably orm of the most important* factors. involved in the success 
of the process was the initial clear definition of role and responsibility 
for tji^ii^ yistitiitiftiij. The recipient* of the project grant vyas the 
cortJnuinity ()rg^i7ation. Tifis organization, through a contractual 
agfl*ement with th^ medical school, "hired'! the training group. THe 
hiisiL tesponsU)ilities o|\»ach organization and of the trainers were 
explicitly outlined. '^fhe raUohale underlying the agreemeht. was the 

# recognition of a need for the^exchange of resources such as funds 
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and specialized skills which would be complementary to the achieve- 
ment of the goals of the«organizations involved. It would also ensure 
the maintenance of the social distance necessary to avoid conflicts 
and the feeling of one organization's being ^'swallowed up** by the 
other (Levine et al. 1963, Lilwak and Meyer 1966). 

The selecticm of evaluation data as a tool for education has proved 
in this instance to. have bdert' an appropriate strategy. The real-life 
nature of the loSi and the relevance of the information for board 
governance were a source of motivation for the learners which 
probably encouraged attendance and Trtggered meaningful discus- 
sions. The subject matter enabled the trainers to remove the **mys- 
tique * surrounding health care delivery and health statistics. Further- 
more, the use of evaluation data has generated a sense of security in 
the board members concerning the decisions they are making; 
indeed, they now wish to extend this approach to other service 
programs for which they are rj^sponsible. 
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DISCUSSION: EDUCATION AND EVALUATION 

David /. ]one$ 



The two papers. dealing with evaluation, providers, and the»com- 
munity stimulated a discussion' to which the .subjects of cojnmunity 
involvement, communication between consumers and providers, and 
the complexity of these activities were central. 

Community involvement was described in its historical-political 
context. One speaker cautioned the group to remember that in the 
Western Hemisphere, community participation predated Columbus- — 
suggesting further that it was civilization that had ruined the process! 
Another speaker characterised political and cultural differences as 
the weak links 'in communication among institutions, tfatir boards, 
and the cornmunity. This, "interrupted communicatipn," as , one 
participant labeled it, underscored the importance of increasing the 
accuracy and objectivity of evaluative data from the cqmmunity, so 
that we might understand the problems better. This observation was 
countered with the question "In any political system, can we|j||/er 
really know how to determine what people want or need?" Neverthe- 
less, there was consensus that constant negotiation was essential in 
detetmining wants and needs. One discussant brought up an even 
more Complicated issue — that community-based boards sometimes 
want to deal with problems which are not amenable to health 
strategies. But one can ask in return, "Who defines /itfa/f/i problems?" 
The discussion was brought full circle by a related observation that 
evaluation (to determine what constitutes a health problem) is 
ultimately based df\ values, and that values are critical to that 
negotiating, communication process. 

Several persons cautioned the confejpnce about the difficulty which 
can arise if we organize around patient o^; consumer needs or wants 
rather than clinical problems — the lack of structurg. Similarly, with^^ 
reference to the paper on provider education, if we derive practice 
from existing providers when the context is not Jcnown or specified, 
we again face the problem of no structure-^ithtr ^for consistent 
delivery or for evaluation. Participants noted the t:onfliGt betvyeen 
dynamism and structure, but no specific resolution was proposed. 

The communication between consumers and providers focused ofi 
the common ground oVi which they frequently meet: the bcferds W 
community oj2}}anizati6ns or institutions. A question frequently im-" 
plied, but not specifically asked, was whether the traditional role of' 
board members has now changed. Histdrically, selection of members 
for community organization boards has been based on specific skills 
or knowledge possessed by. the individual named — the executive, the 
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lawyer, the accountant, the engineer. The other consideration has ' 
been prestige, with board membership reflecting recognition of ' 
position, power, weafth, or past contribution to the community. Ohe - ' 
person commented that the political nature of boards has changed ' 
"from prestige xo work"! ' . ' ^ 

This change in the composition and political nature of cdmn^unity 
boards has intensified communication problems between consumers 
and providers. One participant suggested that there are two lan- 
guages which must be learned and spoken. One* is the "cultural/ 
language of health," reflecting consumers* interesUs in services which 
typify the 'community as they know it; the second, the jargon 'of ■ 
health care. Con.sumers need to learn this "tongue,"-just as providers 
must learn the different cultural languages of health. 

Discussion of the two languages led to another e'xch^^nge concern- 
ing the mystification (through professional education) and th^ demys- 
tification (through consumef^provider communic^ition and fathiliar- 
ization with the jargon) of the health field. There may be an inherent 
threat to professionals in the demystification process^, but that threat 
may be*lessened with an awareness and acceptance that trust is a 
more mutually satisfying response than blind faith. SuVh a view 
suggest that providers can trust consumers when they do riot 
underst^d the^ difference in the cultural language of .heaUh, and 
that consumers can learn to trust providers when they do riot always 
understand the professional jargon of health. ^ * ' 

At heart throughout the discussion was an implicit que^tioa of the l^t- 
risks we are willing to take with each other in trying to achiev^- better 
health for our communities. Perhaps no one summarized ^tter. Jhe ) 
challenge to consumers, providers, community, and commurii^atiori -/ 
than one commentator who suggested that we should join in thi^ riskl 
venture, since, in his words, "We can no longer \ie- the \p^opd. 
^nderstand authenticity.*' ' ^ . 
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SUMMING UP 



Robert }. Haggerty 

^ I would like to begin by making a general point on semantics. We 
have used a lot of words that have different meanings to different 
ones of us. Some of these words are listed in table I. In this whole 
field of evaluation, for the last Vh days, we have included epidemiol- 
ogy, cimical trials, the issue of efficacy, and a series of what could be 
labeled "types" of evaluation, all under the general rubric, evaluation. 
Ur. Yarbrough insisted on the word "assessment" if there was no 
intervention, and on "evaluation" if there was. I think he makes too 
tme a distinction, and I consider the two words as synonymous. ■ 

Within evaluation, we haVe had several types reported, with very 
nice disunctions made between effort, effectiveness (namely impact 
on populations), efficiency, quality, and policy. In addition, some of 
the papers have described the methods themselves. Now to me this is 
a spectrum, and I have tried to classify the papers th'at have been 
presented along this spectrum. Dr. Yarbrough's study was essentially 
a beautiful, classical clinical trial. We had several presentations of the 
effectiveness type (Pless, Ramsey, and Agudelo-Gil). Belville's paper 
was a little more difficult to classify because it was a kind of 
"operations research." It did not have the cost aspect that efficiency 
studies usually have; however, it was on the ef ficiency side. Then we 
had several studies on^methods (Kessner. Martini, and Heredia). 

I found the rather tongue-in-cheek classification that Dr Velasquez^ 
gave us on ev^Muation research very useful, and I have added this to 
table 2 to Illustrate how these other words can be used. I do n6t Wiow 
whether Dr. Velasquez would want to acknowledge the fact Ihat 
academic re.searctv^jjroduces only papers. Government research pro- 
duces pohcyi. providers i»e interested in monitoring operations 
consumers are interested in satisfaction, professional evaluators are 
interested in m^ods, and multisectoral evaluators are concerned 
with change. A^st tl>ia outline put a lot of the words that we have 
bandied about Here intp^iibme sort of perspective, and that may help 
us as we continue. ,> 

Next, I want to mention three .or four major limitations concerning 
evaluation. The first is the lack of methods. Clearly we have had a 
few exciting methods presented to us, but an enormous rtumber of 
problems remain unsolved because we do not have methods for 
solving them. For example, the monitoring of a community's health 
needs is to me one of the high priority items. Yet the methods we 
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Table 1. The Field of Evaluation 



Papers PresinUd 

Epidemiology 

Clinical Triali . ^ Yarbroygh 
Efficacy Studies 

Evaluation-AtseisAaent ^ ^ 

Eflbit » Ramsey. Pless 

Effectiveness Agudck)-GiK 

Efficiency BelvUkr 

Quality . ' Williamson 

Fblicy ■ y Ressner. Martini 

Methods . Heredia 



Table 2. Velasquez Classincation of Health Services Evaluation Research 

' ^1 

Sponsor Product 

Academic Research - Papers 

Government foiicy 
Provider Monitoring of Operations 

Consumer ■. Satisfaction 

Profession of Evaluators Methods 

Mukiaectoral Research _ Change 

• ' : ' ' ' 

used in Rochester to monitor or measure community needs stffSTto 
make the community diagnosis are too expertsive to apply in most 
communities. We need simpler metj^KKls for measiiring these netfds. 
Some researchers suggested that afl we have to do is ask the public 
about community needs. I am not convinced diat one can query only 
the self-appointed leaders jin the community. There has. to be a 
mechanism to reach the disenfranchised and the grol^p that is 
inarticutate about its needs as well. Key informant interviews should 
be based on a representative sample of the public. , 

Secondly, one has to face the trade-offs between different levels of 
soptiisticatioiv in methodology. The academic fesearcher may well 
aim at obuiniri^ a small standard error in his method, yet that may 
not be necessary iri^the operating agency. We must see to what degree 
error can be tolerated 4ji order to make the kinds of decisiohs that we 
need to make. 

The next limitation in the field of , evaluation— and one that is 
clearly made by Dr. Ressner— is the lad^ fcfficacy studies. We really 
have relatively few medical interventions where the data_ on the 
efficacy — that is. on the benefit of any given treatment— are clear. 
Without that information, most evaluation programs must resort to 
measuring the Extent to which people use die services, the ability of 
the community to bear the cost, and the satisfaction of the people. 
These are important issues, but iri the absence of efficacy. I think 
they are very limiting. ^ ^ . ' . 
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The minimaS.role that health services play in producing health is 
another limitation. Dr. Wray emphasized the relation of the economy 
and its changes ttj child health; Dr. Martini also alluded to this 
relationship in his paper. .Unfortunately, we lack good measures to 
indicate those components of health that can be influenced by health 
services. Therefore, in evaluation studies,, we are frequently measure 
ing or trying to measure the outcome of a variety of specific 
treatments that have been put together as a program. 

The lack of cost data presents an additional limitation. I was 
impressed by this lack in the case studfes. Perhaps this is a reflection, 
of my views in 1977 in the United State^, but I have Been guilty of 
ignoring costs for the last 15 years. I think it is very deaf that we Will 
no longer have this- luxury in the'^future'. We are all facing lirtiited 
resources, and evaluation must shoW how wisely these resources are 
being used. At the next conference L would expect to see much more 
sophisticated analysis of the costs' of these programs in dollars, in 
people, and in the trade-offs that the$e dollars and people might buy 
when used in other areas. . 

Finally, the issue of controls or comparison areas came up in 
several of the <liscussion groups. This area is a subset 6f the problems 
of methods. We are not going to have the luxury of random 
assignrnent of patients or programs^to communities in most settings 
or in most ^nedical care studies. We are going to have to use some 
methods of multivariant matching in-order to know whether we have 
a compj^rison group that is similar to the experimental one. The 
sophistication.of that technique has certainly not been advanced very 
much up to now. 

Lest we*" in health services think that we are the only ones with 
these problems, I Would like to quote from ^^n , introduction in a 
recent book on evaluation studies by Dr. Glass (1976). It is rather 
rhetorical in wording, but I hope the message Comes through^. He 
said, * ^ 

The methodologic and philosophical literature on evaluation is 
di^itrcssingly repe'titious. Writers in one field exhume the same dead 
horses that writers in a second interred years before. One who reads 
widely is in jeopardy of gagging from repeated shallowing of dry 
banalities about how evaluation can be distinguished as either 
process or outcome. Qr how relationships between program person- 
nel and evaluators must be handled sensitively. Or'how evaluation 
^Tiust serve decision makers. Scholkrs ia each new area of evaluation 
need not lumber xenophobically up the evolutionary ladder, acting 
as though the problems in their field are so unique that no one can 
pjf)ssibly hav4 addressed them before. " 
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I will probably labor up some of those 'same ladders in this 
discussion by relating some of the practical issues in ^valuation 
research. However, I do not think I will be alone. Those who have 
tried to evaluate evei:ything from^defense to education, to other 
human services have faced the same kind of problems. The concep- 
tual framework for all evaluation is so simple— it is to try and look at ' 
input as related to output or to outcome and to see what intervening 
variables may modify that outcoihe. Workers in the field hav^ made 
a useful distinction between formative and summative evaluation. 
Formative ev<duation is the development of the actual project as you 
g<? along, with feedback to change the program, while summative 
evaluation is an overall evaluation, once the project has been 
completed. By and latge, the problems of evaluation research are not 
in the conceptual framework. That is simple, but there is no doubt ^ 
that we have difficulty carrying out the researdi because of some 
limitationr that I have mentioned before. • * 

• . Now I would like to go through what I consider the seven steps in 
evaluation research and comment on each of these (table 3). The first 
step— to get the program to specify its goals and objectives and its 
underlying assumptions in measurable terms — is an exceedingly • 
difficult one. One of the differentiations that people make betw^een 
other research and evaluation research is that in the latter, one often 
can tell only whether a program has in fact met the objectives sfi>b)^^ 
^ the program. One cannot say whether s^me other method of meetiTT^ 
those objectives would be better. The f)rst step, ^bviously, is to get, 
the program to state its purpose very specifically: The researcjfer may 
want to do something in a different \^ayi but as an evaluator, he must 
have the program say what it wants to achieve. The next step-r- 
describing the program qr defining the independent variable-^is also 
a very difficult task. Whether one inHjlements the program personally 
or whether ene tries to evaluate a program that is iri place, one, of the 
^ery great difficulties is to say specifically, from one place to another, 
what was accotnplished, Hoy^ does ohe control for this independent 
variable? A man who is dbing a considerable amount of evaluation 
research in the, education field recendy told me that the use of onsite 
rec^cjers to determiae what was actually going on in the program, is 
one useful way. He Suggested that we use people who were joindy 
responsible to the program and^o the evaluators to ^escribe what, in 
fact, was delivered by the program and the details of how the 
program was carried outi thereby specifying the independent varia- 
ble. He found'this procedure very useful in evaluating some of the 
Title I educational programs. * 

Ve.ry often the differences that we*find in outcom A between one 
program. and another — when the programs are ostensibly the ^ame — 
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Table 3. Steps in Evaluation Research 



1 . Spcafy goali. objc^vcs. underlyingoassurnpcions of program. 

2. 'Describe program (iiidc^pendcnc variabl^fs). 

; \ 3. Define criteria for goal achievement tdcpcndcrii variables). * 

4. Develop and carry oiit measures of achievement. . ^ 

' 5. Describe intervening variable*— the political and social milieu. 
6. Analyze results* 
^7. Implehient findings. 

are because they have really delivered something different. The 
independent variable is not the same. I might say that when ce tried 
^ this approach in a program in Rochester— putting an onsite person 
in our neighborhood health center to describcS>vhat actually went 
on— there was a strong tendency on the part §f the observer to 
gradually identify with, the staf^l^ She felt increafaing loyalty^to the 
providers, and we lost access to these data. So l art not at aU sanguine 
about getting this kind of material. Vu-..^^ yr^. 

Defining the setting 'in /which the program develops pre)<nts 
' another difficulty in determining appropriate treatjnent. It is not jihl 
-what is delivered, but the whole jmilieu in which it is>lelivered thai 
becomes important in evaluation. The implementation of the U/ 
nationwide Quality Assurance Program— the PSRO Program— isL 
good example of this problem. We will have Spme 200 of these^ 
programs in place: approximately ^half^are already in place. Several 
of us who, were charged with some responsibi^ty for evaluating this 
' program have argued that we needed some kind of planned experi-. 
ment. We had to know^at was beingHone, and we had tb have 
some control over where it was beinj done so that the milieu in whicfT 
/one demonstration was being carried out could be compared to 
another. But we were overruled for political reasons, and the 
program was allowed to develop wherever people had the most 
interest and therefore developed it most rapidly. Under these 
circumstances, it would be foolish to compare ?u:pas lyith and without 
a PSRO. Of course, the. very reasons people (pj^^li^ed these pro- 
^ " ^^^^ their settings very different."^! am reminded of 

if 'v^^"*^ comment th^t one way. to understand the system is to try to 
^^^''^^^^^^ ^ think that is one .way in which we can move fr6m the 
^5^^ of ^hat is going oa to. a more active intervention— 

-W^^^ Dr. Yarbrough defines as evaluation, rather than assessment. 

;;y?^ At the other end of the spectrum is another type of research 

• s^ondary analysis, ^n my opinion, evaluijtion research Ifas suffered 
from too little secondary analysis of data. Most of the large-scale 
programs have an enormous amount- of data thjat' others have 
collected. In our Rochestef study, I kn6w we have data that are 
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untQuched by human thoughts! We-offered. the jcpmputer tapes . to 
people by advertising in Medical Care, . hut v»«J^' fe^v^have ever taken 
up the offer to do secondary analysis.of the tapes' ^hat is part of the 
academic paper business — you have difficulty getting papers pu b- 
4 lished that (Consist of only data froni secondary analysis; However, it 
. is an area where evaluation research could profit from -thf example 
\of some of-.the/Other sciences, such as economics, where secondary 
analysi s hy in fact becotne an honored approach. ^ 

The third step of evaluation research is to define the criteria for 
goal achievernei'u, that is, the dependent variable,^ Perhaps our major 
need here is to develop some proxies for outcome; this need was 
certainly expressed by Dr. Pless in the discussion on chronic disease 
outcomes. It is very difficult, to measure specific functiohs, and there 
are trade-offs between one function ^nd another. I think we must be 
Vjery creative in developing skills to find proxies for outcomes/inkead 
of relying on direct outcomes. 

The next step is to define the intervening variables— the j)pjitka1 
and social milieu. This step has been given too litde emphasis in (his 
meeting. C|ianges in the political' system or social milieuiafftct the 
most carefully planned evaluation. In Boston spme years ago, we 
carried out a very .weir-crontrd[led medical care study, randomly 
assigning families to the new program and watching the control 
groups go through the system in the old fashion. In the midst of this 
study, howe'ver, a local program of great importance — a change in 
the welfare program of reimbursement of the poor— was instituted. 
This change had a great effect oh the utilj^ion of the health services 
by both groups of families. It overwhelmed .the research intery$:nt^iOj[>. 

In Rochester we were monitoring the child health servic^|^^|t:|^^ 
midstudy there occurred the most profound effect on :/k04ri09^:^ 
tthat ,we have experienced: liberalized' abortion in New ^Vjfl^^^i^ 
The number of c^hildjen boril each year dropped by 35^^pei?jE^^ 
j^he first year and 40 percent in the second. The drop in -ffi^ birtb 
rate had far- more influence on child health than did the neighbor- 
hood health centers, the chronic disease programs, or anything else «- 
we were' doing and studying. Similar types of intervening variables 
are very likely to occur. T)ie issue is to have a monitoring' s^i^teni, so 
that when the natural experiment occurs (something you never can 
Vreally plan for),* you can document what has happened atid the real 
effect , of this intervening,variable. 

The next step^. is the analysis of results. Briefly, it is obviously an 
important issue, but we have not given it much attention at this 
conference. However, with the ^modern methods of multivarjant 
regression analysis, one can extract relationships of great importance 
from'data that ort the s'urface;do not seem capable*?)f providing a 
clear answer. ' * , v ' 
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The final step is tjje implementation of findings. We have alluded 
to- this prbcess several times during the course of the, meeting, 
EssentiaUy, evaluation ;is action research, and it srhould lead to some 
ch^rfhge or implementation, i^think we all' agree witfi this statement. 
Therefore, before the project is completect Ve must^eejust how the 
•re3ultsVill be used— or atjeast plan for them. I must say' I have been 
gui4ty of ignoring this step in the past; In a current project we.are 
just starting— the evaluation of child health sefyice^ in the Ury'ted 
States .by setecting a series cff communities and. studying the total 
picture of child health iq those communitie^we are engage$l with' 
the Kennedy School of Public Policy at Harvard, in pJaunin^. for 
impletnentauo;^ of the jjrogranii ahead of time. We plan to reGriiit^a 
faculty m^m^ber jointly with that school, and . his Or hpi'^uha'^y ' 
responsrbility will bethe implementation of the program! ^Xhe plans 
call for a series of meetings 's6 that as the results emerge t^ey v^ll be 
transmitted lo local officials, planning agencies, health departments, 
political groups^and to those responsible for decisions at the national 
level through staff ^meetings with ton^e^sional health workers and 
other people in the political process who are key^to setting up the 
medical care systent in this tountrj-, 1 have no personal experience 
with this type of planned\attempt^ajthpugh members of the political 
sdence and government ^dep^rtmeiW^^ use it quite expertly, 

and it is an important aspect of medical care evaluation research that 
we have ignored in this'copference. , ; ■ - 

lit is difficult to Qeter^hine what programs are worth evaluating. 
Table 4 may giv^ the evalu^tpr s6me guidelines in making his choice. 
Those in charge of the program do need to be able to sute the 
objectives of the program; and these objectives have to be'measurable. 
There must be some testable assumpitions that link the program 
activities 'to the ^ccojmplishment of these objectives. Sometimes you 
run ipto a study group tliat says, "We're doing this; and *it will-result 
In tkis/' and^pu say,-"Well vyhy?'' and the group says, "Well, I 
haven't a clue, yt)u know.'- I think tli^re ought to be some sort qi* ' 
assumption tb^t A will lead to B or, that, A will at least have some 
relationship to B.^ It is most impoVtSfit for thpse in charge of the^ 
pr6gram to^ have some motiv^tjon, ability^nd authority to run the 
program— tjtiat is, to alter the Vnlspurces— and to meet the objectives 
of the program. In 6}her wor^s*, the grpup conducting the program ^ 
♦mifst have some„sort of motivation to change what if ?s doing as a 
result of the evaluation. ^ ^ 

The^e problems I . have raised may make evaluation research • 
appear to be a very risky^ business and one more involved with minor 
adjustments ip ongoing progi;ams than in measurir^ outcortics of real 
importance^;. to health. While this obsei'vation is bfteh true. I also 
bcli^ve^ that cf eativc and innovative efforts in evaluation can lea^ to 
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Fable 4. Guidelines in Selecting Programii for Evaluation 

• . ■ V 

\. Those in chttrgr of u program must sei-objrciivrs thai ar^tmeanirainr. 
2. A program, must havr testable assumptions linking program activities' to accoip- 

pliihment of objectives. ^. 
5. Those in charge must have the incentive, motivation, abilii^? and authority to run 

the program (to alter rcWiq^i^ task, or objectives). ^, 

In tile absence of these» performUiVcr analysts may Imt all (hat is possible since 
overall program impact is not possible. 



significant insight irrtb what works and what does^'pot. It may seem 
that such res'cafch is very ad hoc and empirical, but I thitik the best 
-^f, evaluations can occasionally lead to the development of new 
cones and to major change in human services programs. For 
ainple, our finding of decreased use of services by children at a 
e pf increased availability of services and decreased financial 
friers was to us an unexpected result in Rochester. At least, my 
<?conomi5t friends all look at the outcome and say, "That's wrong; it 
doesn't happen that way." In turn we are led to ask, "Well, what is 
wrpng with the traditional ^eory? yHaVe we found something new 
here?" At least we should be thinkmg ,hov\/^we can feformulate the. 
theoretical basis of the project in evaraatip^ research, and sometimes 
i think that effort is important. * 

I would like to bring up a few a'dditional points in conclusion: the 
relation of evaluatibn to education and the issue of ^alues. 

The relation of evaluation to education. The attitude of how one looks 
at the se|yices that he is being trained to provide is important. I 
personally believe that a critical, questipning view of what heaU\ 
providers can do is an important part of'^edical education. For too 
long we have given the impression that providers are omnipotent and 
thqit we are great Hitalers and savers of mankind. Evaluation research 
can be a very potent aritidote to that impres^on, but there is a danger 
here. The problem is one of balance. I have had pediatric residents 
locfk at their work*so critically that they come away very negalively: 
**Why stay in medicine? Nothing we,d^ does any good." You hz^e to 
weigh this attitude on what I call the doubt/confidence axi^ In' 
dealing with the individual patient, physicians and other providers 
really need to project a kind of confidence — I arwconvinced that is a 
large part of tfee therapeutic benefit. At the same time, physicians 
mu|^ not be allqwed to feel so omnipotent that they never' question 
what good they are doing. In most places, Tthink we have failed to 
achieve that balance between doubt and confidence. Good evali^ation 
research could be a vehicle for teaching that bajjii^jce in medical 
education. . * , , ^ ^ 
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The issue ofvalws. We have had a spirited discussion in one of our 
-groups about values. J think it was a communication problem rather 
than a difference m opinions. Bfut clearly values do dictatt the fact 
that we evaluate a program in the. first place. A lot of people with 
different kmds of values will say the service provided is enoufth of a 
reason for having the service and that evaluation is unnecessag^ The 
solution IS to try to make these value,/ explicit *at the start of any 
■ ['r^M '"Z ""^ speaker-said at the beginning of the conference, we 
Should put the biases out on. the table" at the beginning of^iany 
program. It^y own value is that evaluation research has modest 
contributions to make. We cannot expect evaluation research to tell 
us If we should have a national health program in this country We 
^^annot expect evaluation^ research tp.tell us whether equity of access 
IS a . value that society should seek to achieve. However, once the body 
politic, or indeed bncj. we as individuals in our nonprofessional roles' 
have lobbied for such effect^, (and. it is to be hoped, achieved them). 
...then evaluation research can identify how well.those goals are being 
mci^dnd tpi compare alternative ways to achieve those»froals 
W research is being asked to achieve too much if it is asked 

'"^^^^^S"^.'*- help outline the options, the alternatives, and 

P^^'^'W^'h^ priorities, or af -least the cost of different ways of 
meeting the pj-iqrities. By pladng excessively high expectation* on 
evaluation research,; we ^Iso face the danger of putting too much of 
our resources into evaluation. The trick is to tease out those few 
instances where evaluation can measu!-e something and effect change 
,.That selection ^process /s still an nrt. in my view-one that we can 
•ho^e IS learned by performance. Perhaps conferences like this one 
advance this skijl. In conclusion. I would say that evaluators will not 
.nherit the earth, byt they maylielp to make it a somewhat better-* 
ptacc m whith to live. 
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^ CLOSING REMARKS 

s . ' . 

Antonio Ordonez-Plaja 

The privilege of speaking last has one advantage: The rules of the 
■game are very clear; one iiiust speak loudly in order to be heard, and 
briefly, in .order to be liked. I 'want to be brief, but I cannot let this 
meetirig end without t|^ahking the Fogarty Center and its director for 
having granted us tb^ opportunity for this encounter. I also wish to 
. remember our frijJittl..;'Fred McCrumb. who passed »pw4y recently In 
paying my resjf*^^^? him^ I think I interpret the teelings and 
sentiments df th<iSe;tf'6s who had the good .fortune to meet him. He 
contributed imn^sety to the organization of this conference. We 
shall always cherish aiid respect his meipory. 

The opportunity the last speaker to offer an evaluation of the 
meeting is quite tempting, but it would be premature and would 
constitute an abuse of my privilege. TWo or three observations could 
be made— that there are very few women in attendance and that the 
results of the International Women's \5ear have not been seen here 
Another i« that there are moif Spanish-speaking participants (who 
understand English than there are English-speaking ones who under- 
stand Spanish. ■ ^ 

I believe that the best overall asVssment I could, maiie of the 
meeting is from a prospective of view. WWkn I recall our 

discussions as we made up the agenda. I think we we?ftoo ambitioui 
We included so many themes and wejp*iied so many people wh 
haTT^stic and profound njessages to deliyer that now w^ are a 
leaving ^ith the feeling that our work is unfinished. We are leaving., 
having learned much, but we are still full of doubts, and there are 
problems that we would have liked to discuss-further. The solution 
for this, of course, is to start thinkihg about when we can get together 
again— to plan a second meeting and -not let this meeting be ah 
isolated one. but ^he beginnin-g of a series of communications at the 
hemisphere level. We should not forget Our guests from Great 
Britain, who as usuaf have taught us a .great deal. In further 
meetings, hope .increKiHgtyjo unify, the Norih^the South, and 
the Caribbean. The mdrfe we tqjmmunicate with andlielp each, other, 
the better; it will be for aH'ci^iliy- " • 

ifTwo last thoughts,' we have seen several times, the danger o£ ' 
ey^atiori lies with th^chnician who. -like Alec GuinnesTin "The^ 
Bridge ovey the River W^'-" sometimts falls in love with the object 
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and forgets the objective. Otherwise we are not fulfilling our duty or 
responding to mankind's needs. As *n English statesman said years 
ago when asked to explain why his country had seftous social 
problems, **For me it is very simple; our people ask us for bread, 'and 
we give them statistics." I believe, this failure can be avoided if we 
keep in mind the objectives of any , study or investigation, ^pd the 
first point is the fundamental objective: man, his well-being, arrd hiii 
improvement. Shojuld we ever forget this goal, everything wV do will 
be meaningless, , v • 

Since all of you want to leave, it is unfair for me to take advantage 
of the fact l)iat I have the microphone. Thus I say farewell in the 
words of another great friend, also deceased, Dr. R. Cruishaw, who, 
when he left us at Princeton, said soitiething which he upheld as 
Scottish wisdom, but which I have also heard in very primitive places 
^n other worci^ '*Happy to meet, sorry to part, lucky if we ni^t 
again." 
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Latin America, demographtcjincrft 

tors in, -24 
Ldw birth weight, 28, 29 
and behavioral disturbances, 83 

Malnutrition 
pauses of, 10, 13 
consequences oj, 10* 
prevalence of, 9 ' * ^ 
role in mortality, 8, 25, 
IV^atern^l-Child Care. Model. See 
MCCM. 

Maternal Child ' Care Program 
J (Colombia], 117 
Maternal health, effect on child 
health, 25^26 

Maternal health programs, 29-30 . 
MCCM 

: case classification, 127 
determination of effective de- 
' mand and achievements, 126- * 
127 

estimation of demand fevels, 123- 
125 > 



Index 289 

MCCM(Conl.i 
estimation of mortality and disa- 
bility, 127-128 
e^!li)ation of rei)uirements, 125- 

indices, 128-133 

problem areas, 143-144 

value of, 134, 143 
Medical care, quality of ' 

structure and process criteria, 40 
Migration, effect on healdi, 29 
Monitoring, 62 
Mortality, adolescent, 28 
Morulity, infakit/child 

causes of, 6, 7-8, 13, 15, 25, 73- 
"74 - 

effect of birth weight, 6 
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